Update/improve get deployment elastic

[adrianfusco]

• I wanted to do the SQL equivalent of GROUP BY to take the information of the last build. The way to approach it is using aggregations so I've modified the query used.
• We have been using the scan helper to get the documents from Elasticsearch through a query. In this case it doesn't support the aggregations so in this case we'll have a condition to use the search method of the elastic client.
• We can't send a giant query in the request to the Elasticsearch for asking to all the jobs information so we have been asking the information of the deployment for each job 1:1. Instead of doing one query for job we create a list of jobs sub lists and do calls divided by chunks. chunk_size_for_search quantity will be the size of every sub list. e.g, If we have 2000 jobs we will have the following calls: 2000 / 600 = 3.33 -> 4 calls.
• With method append_get_specific_field we'll get just the specific fields we require from the source field, equivalent to SELECT field in SQL instead of SELECT *.

[bregman-arie]

The difference I see in speed compared to the previous version is amazing. Well done.
One thing we need to address is inconsistency with other sources. I get completely different results running the same queries with Jenkins for example. But this is something that should be addressed in a separate future PR.

Maybe you and @cescgina can take a look at it together and identify any shared components/blocks in the code.

[adrianfusco]

The difference I see in speed compared to the previous version is amazing. Well done. One thing we need to address is inconsistency with other sources. I get completely different results running the same queries with Jenkins for example. But this is something that should be addressed in a separate future PR.

Maybe you and @cescgina can take a look at it together and identify any shared components/blocks in the code.

Thanks. Totally. I've tested now clear; cibyl -vvv -d --source elasticsearch --jobs --topology and I see a good difference:

INFO     cibyl.orchestrator   Took 3.49s to query system osp_jenkins using source elasticsearch of type elasticsearch using method get_deployment

We should take in count the following things:

1. We're not showing all the information in Elasticsearch because we don't have stored all the information of all jobs and builds as Jenkins is doing.
2. Using the Jenkins source we are showing all the jobs. The jobs that don't have information associate we display: No openstack information associated with this job. Using the Elasticsearch source I just show the jobs that have information associated. What do we prefer here?
3. Using Jenkins we're displaying all the information related to each node and role of this one. I don't have this information in Elasticsearch so I can't show it.

[cescgina]

2. Using the Jenkins source we are showing all the jobs. The jobs that don't have information associate we display: No openstack information associated with this job. Using the Elasticsearch source I just show the jobs that have information associated. What do we prefer here?

About this, I worked under the assumption that if the user just passed --jobs --topology without specifying any value for them, then no filtering is done. How does it work for builds in elasticsearch @adrianfusco? If for example called cibyl with --jobs --builds would it show the jobs with no builds?

[adrianfusco]

2. Using the Jenkins source we are showing all the jobs. The jobs that don't have information associate we display: No openstack information associated with this job. Using the Elasticsearch source I just show the jobs that have information associated. What do we prefer here?

About this, I worked under the assumption that if the user just passed --jobs --topology without specifying any value for them, then no filtering is done. How does it work for builds in elasticsearch @adrianfusco? If for example called cibyl with --jobs --builds would it show the jobs with no builds?

Yes, we're showing all the jobs even those with no builds. I've changed the behavior here too.

[adrianfusco]

The difference I see in speed compared to the previous version is amazing. Well done. One thing we need to address is inconsistency with other sources. I get completely different results running the same queries with Jenkins for example. But this is something that should be addressed in a separate future PR.

Maybe you and @cescgina can take a look at it together and identify any shared components/blocks in the code.

I added another change to show all jobs. Let's review it and we can continue working on the other topics in a corespondent task and branch.

[cescgina]

LGTM