Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Visualize: Error: in cell #1: [index_not_found_exception] no such index, with { resource.type="index_or_alias" & resource.id="github_repositories" & index_uuid="na" & index="github_repositories" } #202

Closed
rootsongjc opened this issue Aug 19, 2019 · 19 comments

Comments

@rootsongjc
Copy link

I followed the doc deploying Grimoire from Dockerhub, everything worked smoothly but the Github Repositories had no data with the following messages.

image

Error messages
Visualize: Error: in cell #1: [index_not_found_exception] no such index, with { resource.type="index_or_alias" & resource.id="github_repositories" & index_uuid="na" & index="github_repositories" }

Environment

projects.json

{
    "SOFAStack": {
        "git": [
            "https://github.com/sofastack/sofa-boot",
            "https://github.com/sofastack/sofa-rpc",
            "https://github.com/sofastack/sofa-mosn",
            "https://github.com/sofastack/sofa-tracer",
            "https://github.com/sofastack/sofa-lookout",
            "https://github.com/sofastack/sofa-registry",
            "https://github.com/sofastack/sofa-dashboard",
            "https://github.com/sofastack/sofa-bolt",
            "https://github.com/sofastack/sofa-jraft",
            "https://github.com/sofastack/sofa-acts",
            "https://github.com/sofastack/sofastack.tech"
        ],
        "github": [
            "https://github.com/sofastack/sofa-boot",
            "https://github.com/sofastack/sofa-rpc",
            "https://github.com/sofastack/sofa-mosn",
            "https://github.com/sofastack/sofa-tracer",
            "https://github.com/sofastack/sofa-lookout",
            "https://github.com/sofastack/sofa-registry",
            "https://github.com/sofastack/sofa-dashboard",
            "https://github.com/sofastack/sofa-bolt",
            "https://github.com/sofastack/sofa-jraft",
            "https://github.com/sofastack/sofa-acts",
            "https://github.com/sofastack/sofastack.tech"
        ]
    },
    "alipay": {
        "git": [
            "https://github.com/alipay/SoloPi"
        ],
        "github": [
            "https://github.com/alipay/SoloPi"
        ]
    },
    "seata": {
        "git": [
            "https://github.com/seata/seata"
        ],
        "github": [
            "https://github.com/seata/seata"
        ]
    },
    "SQLFlow": {
        "git": [
            "https://github.com/sql-machine-learning/sqlflow"
        ],
        "github": [
            "https://github.com/sql-machine-learning/sqlflow"
        ]
    }
}
@valeriocos
Copy link
Member

Hi @rootsongjc ,

Can you share the setup.cfg? it may be possible that in the section panels the param github-repos is set True?

@rootsongjc
Copy link
Author

rootsongjc commented Aug 19, 2019

I can't find the setup.cfg, do you mean the dashboard.cfg?

dashboard.cfg
I just keep it like the original as the docker image.

# Mordred configuration file (Dashboard)
# Parameters related to the dashboard and how it is produced
# This is usually updated by the person maintaining the dashboard
#
# List: [val1, val2 ...]
# Int: int_value
# Int as string: "Int"
# List as string: "[val1, val2 ...]"
# String: string_value
# None: None, none
# Boolean: true, True, False, false

[general]
# Update incrementally, forever
update = true
# Don't start a new update earlier than (since last update, seconds)
min_update_delay = 300
# Produce debugging data for the logs
debug = true

[es_enrichment]
# Refresh identities and projects for all items after enrichment
autorefresh = true

[sortinghat]
# Run affilation
affiliate = True
# How to match to unify
matching = [email]
# How long to sleep before running again, for identities tasks
sleep_for = 100

[panels]
# Dashboard: default time frame
kibiter_time_from = "now-1y"
# Dashboard: default index pattern
kibiter_default_index = "git"
# GitHub repos panels
github-repos = true

[phases]
collection = true
identities = true
enrichment = true
panels = true

[git]
# Names for raw and enriched indexes
raw_index = git_grimoirelab-raw
enriched_index = git_grimoirelab
studies = [enrich_demography:git, enrich_areas_of_code:git, enrich_onion:git]

[github]
# Names for raw and enriched indexes
raw_index = github_grimoirelab-raw
enriched_index = github_grimoirelab
# Sleep it GitHub API rate is exhausted, waited until it is recovered
sleep-for-rate = true

[pipermail]
# Names for raw and enriched indexes
raw_index = mail_grimoirelab-raw
enriched_index = mail_grimoirelab

# Studies

[enrich_demography:git]
#no_incremental = true   # default: false

[enrich_areas_of_code:git]
in_index = git_grimoirelab-raw
out_index = git_aoc_grimoirelab-enriched
#sort_on_field = metadata__timestamp
#no_incremental = false

[enrich_onion:git]
in_index = git_grimoirelab
out_index = git_onion_grimoirelab-enriched
#data_source = git
#contribs_field = hash
#timeframe_field = grimoire_creation_date
#sort_on_field = metadata__timestamp
#no_incremental = false

I want to figure out why these errors came out because I want to monitor the GitHub repos.

@valeriocos
Copy link
Member

valeriocos commented Aug 19, 2019

Thank you for sharing it. Try to modify your setup.cfg and projects.json in the following way:

setup.cfg

[github:repo]
raw_index = github_grimoirelab_stats-raw
enriched_index = github_grimoirelab_stats
category = repository
no-archive = true
sleep-for-rate = true

projects.json

{
    "SOFAStack": {
        "git": [
            "https://github.com/sofastack/sofa-boot",
            "https://github.com/sofastack/sofa-rpc",
            "https://github.com/sofastack/sofa-mosn",
            "https://github.com/sofastack/sofa-tracer",
            "https://github.com/sofastack/sofa-lookout",
            "https://github.com/sofastack/sofa-registry",
            "https://github.com/sofastack/sofa-dashboard",
            "https://github.com/sofastack/sofa-bolt",
            "https://github.com/sofastack/sofa-jraft",
            "https://github.com/sofastack/sofa-acts",
            "https://github.com/sofastack/sofastack.tech"
        ],
        "github": [
            "https://github.com/sofastack/sofa-boot",
            "https://github.com/sofastack/sofa-rpc",
            "https://github.com/sofastack/sofa-mosn",
            "https://github.com/sofastack/sofa-tracer",
            "https://github.com/sofastack/sofa-lookout",
            "https://github.com/sofastack/sofa-registry",
            "https://github.com/sofastack/sofa-dashboard",
            "https://github.com/sofastack/sofa-bolt",
            "https://github.com/sofastack/sofa-jraft",
            "https://github.com/sofastack/sofa-acts",
            "https://github.com/sofastack/sofastack.tech"
        ],
        "github:repo": [
            "https://github.com/sofastack/sofa-boot",
            "https://github.com/sofastack/sofa-rpc",
            "https://github.com/sofastack/sofa-mosn",
            "https://github.com/sofastack/sofa-tracer",
            "https://github.com/sofastack/sofa-lookout",
            "https://github.com/sofastack/sofa-registry",
            "https://github.com/sofastack/sofa-dashboard",
            "https://github.com/sofastack/sofa-bolt",
            "https://github.com/sofastack/sofa-jraft",
            "https://github.com/sofastack/sofa-acts",
            "https://github.com/sofastack/sofastack.tech"
        ]
    },
    "alipay": {
        "git": [
            "https://github.com/alipay/SoloPi"
        ],
        "github": [
            "https://github.com/alipay/SoloPi"
        ],
        "github:repo": [
            "https://github.com/alipay/SoloPi"
        ]
    },
    "seata": {
        "git": [
            "https://github.com/seata/seata"
        ],
        "github": [
            "https://github.com/seata/seata"
        ],
       "github:repo": [
            "https://github.com/seata/seata"
        ]
    },
    "SQLFlow": {
        "git": [
            "https://github.com/sql-machine-learning/sqlflow"
        ],
        "github": [
            "https://github.com/sql-machine-learning/sqlflow"
        ],
        "github:repo": [
            "https://github.com/sql-machine-learning/sqlflow"
        ]
    }
}

I want to figure out why these errors came out because I want to monitor the GitHub repos.

Sorry, the documentation is problaby not good enough.
If the param github-repos = true is set to True (panels section of the setup.cfg), the dashboards to show repo statistics will be uploaded to Kibiter. In this case, github:repo sections should be included in the projects.json and in the setup.cfg.

I will update the doc at: https://github.com/chaoss/grimoirelab-sirmordred/blob/master/README.md#panels, to make it explict that github:repo sections should be declared if github-repos = true is set to True

@valeriocos
Copy link
Member

Feel free to close the issue @rootsongjc if the problem is solved, and don't hesitate to leave comments on the referenced PR if you think it's missing something. Thanks

@rootsongjc
Copy link
Author

@valeriocos Thank you, I am working on it, if I have any new progress, I will leave a comment here.

@rootsongjc
Copy link
Author

rootsongjc commented Aug 19, 2019

@valeriocos I didn't find the setup.cfg file. Here are my config files.

dashboard.cfg

# Mordred configuration file (Dashboard)
# Parameters related to the dashboard and how it is produced
# This is usually updated by the person maintaining the dashboard
#
# List: [val1, val2 ...]
# Int: int_value
# Int as string: "Int"
# List as string: "[val1, val2 ...]"
# String: string_value
# None: None, none
# Boolean: true, True, False, false

[general]
# Update incrementally, forever
update = true
# Don't start a new update earlier than (since last update, seconds)
min_update_delay = 300
# Produce debugging data for the logs
debug = true

[es_enrichment]
# Refresh identities and projects for all items after enrichment
autorefresh = true

[sortinghat]
# Run affilation
affiliate = True
# How to match to unify
matching = [email]
# How long to sleep before running again, for identities tasks
sleep_for = 100

[panels]
# Dashboard: default time frame
kibiter_time_from = "now-1y"
# Dashboard: default index pattern
kibiter_default_index = "git"
# GitHub repos panels
github-repos = true

[phases]
collection = true
identities = true
enrichment = true
panels = true

[git]
# Names for raw and enriched indexes
raw_index = git_grimoirelab-raw
enriched_index = git_grimoirelab
studies = [enrich_demography:git, enrich_areas_of_code:git, enrich_onion:git]

[github]
# Names for raw and enriched indexes
raw_index = github_grimoirelab-raw
enriched_index = github_grimoirelab
# Sleep it GitHub API rate is exhausted, waited until it is recovered
sleep-for-rate = true

[github:repo]
raw_index = github_grimoirelab_stats-raw
enriched_index = github_grimoirelab_stats
category = repository
no-archive = true
sleep-for-rate = true

[pipermail]
# Names for raw and enriched indexes
raw_index = mail_grimoirelab-raw
enriched_index = mail_grimoirelab

# Studies

[enrich_demography:git]
#no_incremental = true   # default: false

[enrich_areas_of_code:git]
in_index = git_grimoirelab-raw
out_index = git_aoc_grimoirelab-enriched
#sort_on_field = metadata__timestamp
#no_incremental = false

[enrich_onion:git]
in_index = git_grimoirelab
out_index = git_onion_grimoirelab-enriched
#data_source = git
#contribs_field = hash
#timeframe_field = grimoire_creation_date
#sort_on_field = metadata__timestamp
#no_incremental = false

infra.cfg

# Mordred configuration  (Infra)
# Parameters related to the infrastructure (services, directories...)
# This is updated usually by the person maintaining the infrastructure
#
# Version for grimoirelab/install & grimoirelab/full containers
# Servers in localhost, default ports, credentials in grimoirelab/full
#
# List: [val1, val2 ...]
# Int: int_value
# Int as string: "Int"
# List as string: "[val1, val2 ...]"
# String: string_value
# None: None, none
# Boolean: true, True, False, false

[general]
# Directory for logs
logs_dir = /logs
# File with aliases definitions
aliases_file = /aliases.json
# File with menu defintions
menu_file = /menu.yaml

[projects]
projects_file = /projects.json

[es_collection]
# Elasticsearch for data collection (raw indexes)
url = http://localhost:9200
user =
password =

[es_enrichment]
# Elasticsearch for enriched data
url = http://localhost:9200
user =
password =

[sortinghat]
# Infrastructure for SortingHat (MariaDB/MySQL database)
host = localhost
user = grimoirelab
password = grimoirelab
database = grimoirelab_sh

# Use organizations file
load_orgs = false
orgs_file = /orgs.json

# Identities file in GrimoireLab format
identities_file = [/identities.yaml]
identities_format = grimoirelab

[panels]
# Kibitter / kibana entry point
kibiter_url = http://localhost:5601

project.cfg

[general]
short_name = CHAOSS

[sortinghat]
# Use organizations file
load_orgs = false
orgs_file = orgs.json

# Organization name for people not affiliated to any organization
unaffiliated_group = Unknown

# Ids known to be bots
bots_names = [Beloved Bot]

# How to autoprofile
autoprofile = [CHAOSS:manual,git,github]

Every time I open http://localhost:5601, the link below repositories didn't appear.

image

github_repositories index has no data.

image

@valeriocos
Copy link
Member

@valeriocos I didn't find the setup.cfg file. Here are my config files.

The dashboard.cfg should be fine.

Can you share the output of:

@rootsongjc
Copy link
Author

curl http://localhost:9200/_cat/indices
yellow open github_grimoirelab-raw         NQdR4Y93Q86mnmtqszp5QA 5 1   5293    23  33.6mb  33.6mb
yellow open git_grimoirelab-raw            za01Kt3_RnqLvhiMQhLXGw 5 1  12693    14  30.3mb  30.3mb
yellow open git_onion_grimoirelab-enriched zipnNtfJTy6NVDSw1dxEHQ 5 1   5234     0   1.5mb   1.5mb
yellow open github_grimoirelab_stats-raw   6wXS_FyNT92Qu9SAMy2s3g 5 1     88     0   2.2mb   2.2mb
yellow open github_grimoirelab             _yJOKgVZT0GQvycjSMorUA 5 1   5288  4267  11.6mb  11.6mb
yellow open git_grimoirelab                78FFpON0Ty6G6eRGjQBzcA 5 1  12693    51  48.3mb  48.3mb
yellow open github_grimoirelab_stats       _dVnoqaZRpCS-jA1rI--BA 5 1     88     8 513.6kb 513.6kb
yellow open .kibana                        obJQwYoCSo-_8OuIIOpW5g 1 1    199     3   242kb   242kb
yellow open git_aoc_grimoirelab-enriched   v0xGBHIWSu-rdWjWxbhX4Q 5 1 187461 73484 198.7mb 198.7mb
curl http://localhost:9200/_alias?pretty
{
  "github_grimoirelab-raw" : {
    "aliases" : {
      "github-raw" : { }
    }
  },
  "git_grimoirelab-raw" : {
    "aliases" : {
      "git-raw" : { }
    }
  },
  "git_onion_grimoirelab-enriched" : {
    "aliases" : {
      "all_onion" : { }
    }
  },
  "github_grimoirelab_stats-raw" : {
    "aliases" : { }
  },
  "github_grimoirelab" : {
    "aliases" : {
      "github_issues" : { },
      "github_issues_enrich" : { },
      "issues_closed" : { },
      "issues_created" : { },
      "issues_updated" : { }
    }
  },
  "git_grimoirelab" : {
    "aliases" : {
      "affiliations" : { },
      "demographics" : { },
      "git" : { },
      "git_author" : { },
      "git_enrich" : { }
    }
  },
  "github_grimoirelab_stats" : {
    "aliases" : {
      "github" : { }
    }
  },
  ".kibana" : {
    "aliases" : { }
  },
  "git_aoc_grimoirelab-enriched" : {
    "aliases" : {
      "git_areas_of_code" : { }
    }
  }
}

@valeriocos
Copy link
Member

valeriocos commented Aug 19, 2019

Thank you for sharing.
There is data about github repo stats (github_grimoirelab_stats), however the alias on that index is not correctly set (I'll check why).
To fix your problem, try the following:

  • In kibiter, click on the wrench on the left, a console will appear
  • write and execute the following instructions:
POST /_aliases
{
    "actions" : [
        { "add" : { "index" : "github_grimoirelab_stats", "alias" : "github_repositories" } }
    ]
}

this will add the alias github_repositories to the index github_grimoirelab_stats

POST /_aliases
{
    "actions" : [
        { "remove" : { "index" : "github_grimoirelab_stats", "alias" : "github" } }
    ]
}

this will remove the alias github from the index github_grimoirelab_stats

you should end up with something like this:
captura_31

To make sure the modifications on the aliases took place, execute again the command curl http://localhost:9200/_alias?pretty

@rootsongjc
Copy link
Author

@valeriocos It works, thank you. I think we should update the Grimoirelab Tutorial to guide users on how to monitor GitHub repositories correctly.

@valeriocos
Copy link
Member

Thank you for your feeback, it would be great if you could contribute to the tutorial (@jgbarah can help you on this).

@antonionardella
Copy link

Hello,

I got a similar error with the grimoire/full docker image:

Timelion: Error: in cell #1: [index_not_found_exception] no such index, with { resource.type="index_or_alias" & resource.id="github_repositories" & index_uuid="na" & index="github_repositories" }

image

This is because my resource.id is different.

Here the result of curl http://localhost:9200/_cat/indices

yellow open github_iota_stats-raw        079vhlrUT9OGv7woKdhXtQ 5 1   882     0   9.3mb   9.3mb
yellow open .kibana                      y-jpOQITTYasZ5WvDUl5Rw 1 1   247    47 295.6kb 295.6kb
yellow open github_iota_raw              alWQyd0JQu-b5_Crrl1Q3A 5 1  7388    23  33.7mb  33.7mb
yellow open github_iota_enriched         IiW356vNS5Sab1aTJGlb9g 5 1  7388  2129  30.9mb  30.9mb
yellow open discourse_iota_raw           8i_-vzIJQ9W75Y9vVv3r2A 5 1    77     1   1.6mb   1.6mb
yellow open git_iota_enriched            56dRuf_ORV60yFQdiPdcUw 5 1 64089 26445 131.3mb 131.3mb
yellow open stackexchange_iota_enriched  FJTtmgbFTFumFZf50J_APQ 5 1     0     0   1.2kb   1.2kb
yellow open discourse_iota_enriched      9mZO79P8SZe9sWc0c0oD_A 5 1   516    32   760kb   760kb
yellow open github_iota_stats            0Cx3SZOvT2yYELJDDyw6WA 5 1   784    12 602.6kb 602.6kb
yellow open twitter_iota_raw             hIEzyXZCS5qucGUhqvCY1Q 5 1  2364  2088  11.7mb  11.7mb
yellow open git_iota_raw                 Y-fEAWyuTmmzgUci0qZ1aw 5 1 64089     0  86.8mb  86.8mb
yellow open git_aoc_grimoirelab-enriched ctEVGhOtRt6-tHRyJyx7RA 5 1     0     0   1.1kb   1.1kb
yellow open twitter_iota_enriched        YdNo5z-qQQydst5Rqgic_w 5 1  2364  2109  12.4mb  12.4mb
yellow open stackexchange_iota_raw       rVzK0PzVRcuUq1KHVyDqhA 5 1     0     0   1.2kb   1.2kb

As soon as I changed the alias with:

POST /_aliases
{
    "actions" : [
        { "add" : { "index" : "github_iota_stats", "alias" : "github_repositories" } }
    ]
}

It started working again:
image

Here my redacted credentials.cfg

[github]
api-token = [REDACTED]
raw_index = github_iota_raw
enriched_index = github_iota_enriched

[git]
raw_index = git_iota_raw
enriched_index = git_iota_enriched

[github:repo]
api-token = [REDACTED]
raw_index = github_iota_stats-raw
enriched_index = github_iota_stats
category = repository
no-archive = true
sleep-for-rate = true

[twitter]
api-token  = [REDACTED]
raw_index = twitter_iota_raw
enriched_index = twitter_iota_enriched

[stackexchange]
raw_index = stackexchange_iota_raw
enriched_index = stackexchange_iota_enriched
api-token = [REDACTED]
tag =

[discourse]
raw_index = discourse_iota_raw
enriched_index = discourse_iota_enriched

Is is possible that the resource.id is not read correctly? Where is it set?
Thank you.

@valeriocos
Copy link
Member

Hi @ioiobzit, thanks for reporting this problem.

The aliases are declared at https://github.com/chaoss/grimoirelab-sirmordred/blob/master/aliases.json (note that the format has recently changed). For every raw/enriched index, the corresponding aliases are set by this code:
https://github.com/chaoss/grimoirelab-elk/blob/ac980762f50e3280822803326d724a6b6694912f/grimoire_elk/elastic.py#L82

I'll try to replicate the behaviour you report (the alias for github:repo is not set automatically). In the meanwhile, the solution is to set it manually (as you did).

@BlairLee
Copy link

BlairLee commented Oct 29, 2020

Hi I also met this issue when I use docker-compose up. I want to add dockerhub and github:repo but got this error:
Screen Shot 2020-10-29 at 10 41 57 AM

Could you please tell me what I should do? Here are my config files:
setup.cfg:

[general]
short_name = GrimoireLab
update = true
min_update_delay = 60
debug = false
logs_dir = /home/bitergia/logs
aliases_file = /home/bitergia/conf/aliases.json

[projects]
projects_file = /home/bitergia/conf/projects.json

[es_collection]
url = http://elasticsearch:9200

[es_enrichment]
url = http://elasticsearch:9200
autorefresh = true

[sortinghat]
host = mariadb
user = root
password =
database = demo_sh
load_orgs = true
orgs_file = /home/bitergia/conf/organizations.json
autoprofile = [github, pipermail, git]
matching = [email]
sleep_for = 100
unaffiliated_group = Unknown
affiliate = true
strict_mapping = false
reset_on_load = false
identities_file = [/home/bitergia/conf/identities.yml]
identities_format = grimoirelab

[panels]
kibiter_time_from = now-5y
kibiter_default_index = git
kibiter_url = http://kibiter:5601
kibiter_version = 6.1.4-1
github-repos = true

[phases]
collection = true
identities = true
enrichment = true
panels = true

#[bugzillarest]
#raw_index = bugzillarest_demo_raw
#enriched_index = bugzillarest_demo_enriched
#no-archive = true

#[confluence]
#no-archive = true
#raw_index = confluence_demo_raw
#enriched_index = confluence_demo_enriched

[git]
raw_index = git_demo_raw
enriched_index = git_demo_enriched
latest-items = true
studies = [enrich_demography:git, enrich_areas_of_code:git, enrich_onion:git]

#[github]
#api-token = <YOUR_API_TOKEN_WHERE>
#enterprise-url = <YOUR_GITHUB_ENTERPRISE_URL>
#raw_index = github_demo_raw
#sleep-for-rate = true
#sleep-time = "300"
#enriched_index = github_demo_enriched

#[gitlab:issues]
#api-token = <YOUR_API_TOKEN_HERE>
#raw_index = gitlab_issues_demo_raw
#enriched_index = gitlab_issues_demo_enriched
#no-archive = true
#enterprise-url = <YOUR_GITLAB_INSTANCE_URL>
#sleep-for-rate = true

#[gitlab:merge]
#api-token = <YOUR_API_TOKEN_HERE>
#raw_index = gitlab_merges_demo_raw
#enriched_index = gitlab_merges_demo_enriched
#no-archive = true
#enterprise-url = <YOUR_GITLAB_INSTANCE_URL>
#category = merge_request
#sleep-for-rate = true

#[jira]
#raw_index = jira_demo_raw
#enriched_index = jira_demo_enriched
#no-archive = true

#[pipermail]
#raw_index = pipermail_demo_raw
#enriched_index = pipermail_demo_enriched
#no-verify = true

#[mediawiki]
#raw_index = mediawiki_demo_raw
#enriched_index = mediawiki_demo_enriched
#no-archive = true

#[meetup]
#raw_index = meetup_demo_raw
#enriched_index = meetup_demo_enriched
#api-token = <YOUR_API_TOKEN_WHERE>
#no-archive = true
#sleep-for-rate = true
#sleep-time = "300"

#[stackexchange]
#raw_index = stackexchange_demo_raw
#enriched_index = stackexchange_demo_enriched
#api-token = <YOUR_API_TOKEN_WHERE>
#no-archive = true

#[slack]
#raw_index = slack_demo_raw
#enriched_index = slack_demo_enriched
#api-token = <YOUR_API_TOKEN_WHERE>
#no-archive = true

#[supybot]
#raw_index = supybot_demo_raw
#enriched_index = supybot_demo_enriched

#[twitter]
#raw_index = twitter_demo_raw
#enriched_index = twitter_demo_enriched
#api-token = <YOUR_API_TOKEN_WHERE>
#no-archive = true
#sleep-for-rate = true
#sleep-time = 300

## studies based on enriched indexes

[enrich_demography:git]

[enrich_areas_of_code:git]
in_index = git_demo_raw
out_index = git-aoc_demo_enriched

[enrich_onion:git]
in_index = git
out_index = git-onion_demo_enriched
contribs_field = hash

[github:repo]
raw_index = github_grimoirelab_stats-raw
enriched_index = github_grimoirelab_stats
category = repository
no-archive = true
sleep-for-rate = true

[dockerhub]
raw_index = dockerhub_raw
enriched_index = dockerhub_enriched
no-archive = true (suggested)

projects.json

{
  "datasetinsights": {
    "meta": {
      "title": "Dataset Insights"
    },
    "git": [
      "https://github.com/Unity-Technologies/datasetinsights"
    ],
    "github": [
      "https://github.com/Unity-Technologies/datasetinsights"
    ],
    "github:repo": [
      "https://github.com/Unity-Technologies/datasetinsights"
    ],
    "dockerhub": [
      "unitytechnologies datasetinsights"
    ]
  },
  "perception": {
    "meta": {
      "title": "Perception SDK"
    },
    "git": [
      "https://github.com/Unity-Technologies/com.unity.perception"
    ],
    "github": [
      "https://github.com/Unity-Technologies/com.unity.perception"
    ],
    "github:repo": [
      "https://github.com/Unity-Technologies/com.unity.perception"
    ]
  }
}

After I edit aliases, I got this error on console:
Screen Shot 2020-10-29 at 11 15 12 AM

This is the results after I ran curl http://localhost:9200/_cat/indices:

green  open .kibana_2               Zczqpp12TAe7rBAFQbhPgQ 1 0   218    4 267.5kb 267.5kb
yellow open git-onion_demo_enriched 9ZFy3YdHQ8yb7XkRaXCpxw 5 1   234    0 151.1kb 151.1kb
yellow open git_demo_enriched       OnnYIaKjRZKonSn7mg8gFg 5 1   983    0   3.5mb   3.5mb
yellow open git_demo_raw            ZUZCSdyoRT-QhRWJt199JA 5 1   983    0   2.3mb   2.3mb
yellow open .grimoirelab-sigils     TT-wZDtoTpW2B0lpA_s-Xw 5 1    37    2  89.1kb  89.1kb
green  open .kibana_5               LsAWG6srS4KujcAVZNTNTg 1 0   218    3 286.3kb 286.3kb
yellow open git-aoc_demo_enriched   p3c44G3-SUOB_AwokiX6hQ 5 1 12764 4170  11.9mb  11.9mb
green  open .kibana_10              Wy_9MJJ7QSugCcZih3-3BA 1 0   232    3 292.8kb 292.8kb
green  open .kibana_6               p94FojwKRoujEIcSZpgphQ 1 0   218    3 286.3kb 286.3kb
green  open .kibana_11              kDfpD8FFSBK9T3SE0f9ZHg 1 0   232    3 292.7kb 292.7kb
green  open .kibana_1               499WrTU3SiaPBK6pkTyy-Q 1 0   210   24   327kb   327kb
green  open .kibana_9               LyJ8RELPRrCuwFlFOyXA6w 1 0   232    3 292.8kb 292.8kb
green  open .kibana_4               xuDHwjhIQcWxG56KZb5S7Q 1 0   218    3 286.4kb 286.4kb
green  open .kibana_8               FeKTPlyHTlSXOvU5KMOdaw 1 0   232    3 304.1kb 304.1kb
green  open .kibana_7               SU3Fx5rRQjW6GFglzU-6mQ 1 0   218    3 286.4kb 286.4kb
green  open .kibana_3               37gI5Gr8RySUKLjSn-6Lmw 1 0   218    3 286.4kb 286.4kb

@zhquan
Copy link
Member

zhquan commented Oct 30, 2020

Hi @BlairLee,

The error you see on the dashboard is that the index patterns don't exist, you can use kidash to import those index patterns.

But the param panels is activated on the section [phases] so it will upload automatically if the setup.cfg is correct.

You can't add aliases to an index if that index doesn't exist. Those indexes are not created so I guess the mordred container is not running or the container failure due to the incorrect setup.cfg file.

Could you remove (suggested) on the section [dockerhub] and try again?

Best,
Quan

@BlairLee
Copy link

BlairLee commented Nov 2, 2020

I removed (suggested) on [dockerhub], but I still encountered the index issue and got this error on console:

mordred_1        | Traceback (most recent call last):
mordred_1        |   File "/usr/local/bin/sirmordred", line 118, in <module>
mordred_1        |     logger = setup_logs(logs_dir, debug_mode)
mordred_1        |   File "/usr/local/bin/sirmordred", line 61, in setup_logs
mordred_1        |     fh = logging.FileHandler(fh_filepath)
mordred_1        |   File "/usr/lib/python3.5/logging/__init__.py", line 1009, in __init__
mordred_1        |     StreamHandler.__init__(self, self._open())
mordred_1        |   File "/usr/lib/python3.5/logging/__init__.py", line 1038, in _open
mordred_1        |     return open(self.baseFilename, self.mode, encoding=self.encoding)
mordred_1        | PermissionError: [Errno 13] Permission denied: '/home/bitergia/logs/all.log'

Here is my setup.cfg:

[general]
short_name = GrimoireLab
update = true
min_update_delay = 60
debug = false
logs_dir = /home/bitergia/logs
aliases_file = /home/bitergia/conf/aliases.json

[projects]
projects_file = /home/bitergia/conf/projects.json

[es_collection]
url = http://elasticsearch:9200

[es_enrichment]
url = http://elasticsearch:9200
autorefresh = true

[sortinghat]
host = mariadb
user = root
password =
database = demo_sh
load_orgs = true
orgs_file = /home/bitergia/conf/organizations.json
autoprofile = [github, pipermail, git]
matching = [email]
sleep_for = 100
unaffiliated_group = Unknown
affiliate = true
strict_mapping = false
reset_on_load = false
identities_file = [/home/bitergia/conf/identities.yml]
identities_format = grimoirelab

[panels]
kibiter_time_from = now-5y
kibiter_default_index = git
kibiter_url = http://kibiter:5601
kibiter_version = 6.1.4-1
github-repos = true

[phases]
collection = true
identities = true
enrichment = true
panels = true

[dockerhub]
raw_index = dockerhub_raw
enriched_index = dockerhub_enriched
no-archive = true

[git]
raw_index = git_raw
enriched_index = git_enriched
latest-items = true

[github:repo]
raw_index = github_stats-raw
enriched_index = github_stats_enriched
category = repository
no-archive = true
sleep-for-rate = true

@zhquan
Copy link
Member

zhquan commented Mar 3, 2021

Hi @BlairLee,

Sorry for the late reply,

This new error is caused by the permission of your directories. Docker cannot create the file all.log

If you are using docker-compose as https://github.com/chaoss/grimoirelab/blob/master/docker-compose/docker-compose.yml#L56 make sure that your docker user has read and write permission on that directory /tmp/ in this example, but you can create another directory. If you don't know which user is using your docker, create a new directory and assign it 777.

example:

$ mkdir ~/mordred-log
$ chmod 777 ~/mordred-log

In your docker-compose.yml

    mordred:
      restart: on-failure:5
      image: bitergia/mordred:latest
      volumes:
         ...
         ...
        - /home/NAME/mordred-log:/home/bitergia/logs

And restart the container.

I hope it helps you.

Best,
Quan

@jakspok
Copy link

jakspok commented Feb 10, 2022

Hi @BlairLee,

How are you, I went through the same error and the solution that worked for me running compose on win 10 was that within the docker-compose.yml configuration for the line where the configuration is declared
mordred:
restart: on-failure:5
image: bitergia/mordred:latest
volumes:
- /tmp/:/home/bitergy/logs

change it for this

  - ../home/bitergia/logs

with which the compose runs perfectly

I hope I've helped

@yodayuuki55
Copy link

yodayuuki55 commented Feb 14, 2022 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants