Opensearch data sourcing

[vancauwe]

Dear community,

Thank you for the impressive GrimoireLab tooling!
We are trying to get GrimoireLab running for our Swiss Data Science Center & EPFL hackathon in 1 week and a half.

We have gone over all the docs and followed procedures listed here, adding our projects to project.json and adding API tokens and configurations to setup.cfg. We can see Mordred "processing" the repositories defined in projects.json and we see all gets collected with the "collection finished" logs. We can also import template dashboards.

But on connecting to OpenSearch we are confused: where do we visualise for the githubs specified in project.json ? we can create dashboard but our Data Sources are empty.

We have read the docs and followed all instructions here:

• https://github.com/chaoss/grimoirelab/blob/main/README.md
• https://github.com/chaoss/grimoirelab/blob/main/docker-compose/README.md (as well as the docs on setup.cfg and projects.json)

The docs have a lot of legacy documentation which make them hard to navigate, so help would really be appreciated. Do you have any example tutorial video ? It would be great to just see docker compose spun up and a quick demo of how to add a project from GitHub and visualise it with Sigils dashboard.

Thank you so much !

[vancauwe]

@jjmerchante, @Nebrethar, @sduenas , , highlighting this post. Thank you so much !

[jjmerchante]

Hi @vancauwe,

In OpenSearch, make sure the indices are created and contain data. You can access OpenSearch directly from
https://localhost:9200/_cat/indices?v

The docs.count column should be a number bigger than 0. For example, for Git data:

https://localhost:9200/_cat/indices?v
health status index                        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   git-aoc_demo_enriched        Njn0j94vTzGfOVxIhxNG0g   1   1       5070        10140      6.7mb          6.7mb
yellow open   git_demo_enriched            gewsdsShRW2QtGv-kiTWTQ   1   1        943         4095      9.4mb          9.4mb
yellow open   git-onion_demo_enriched      vLUctLepQ6iTGRrtuzRFgQ   1   1        672            0    142.2kb        142.2kb
yellow open   git_demo_raw                 yWB5ONA4TpyIDriQ7cGJrA   1   1        943            0      1.3mb          1.3mb
...

If your _enriched index contains documents, open OpenSearch Dashboards (e.g., http://localhost:8000) and go to Discover in the left menu.

If a window appears for creating an index pattern, create a new one. For GitHub, use github_pull_requests or github_issues. Then, select grimoire_creation_date as the time field for better results.

If you already have an index pattern in the Discover tab, select it. In the top right corner, choose a large time range, such as "Last 20 years," to see all historical data.

If you see data there, you can create any dashboard from it.

Do you see any specific error?

[vancauwe]

Thank you @jjmerchante , we will be giving it a whirl beginning of January and will reach out to you if we have further questions.