Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GSoC idea: Reporting of CHAOSS Metrics #70

Closed
jgbarah opened this issue Feb 16, 2018 · 81 comments
Closed

GSoC idea: Reporting of CHAOSS Metrics #70

jgbarah opened this issue Feb 16, 2018 · 81 comments

Comments

@jgbarah
Copy link
Contributor

jgbarah commented Feb 16, 2018

[ This issue for addressing questions and comments related to this GSoC idea, which is one of the ideas proposed by the CHAOSS group for the 2018 edition of GSoC ]

[Edited, 2018-03-03: Added procedure for uploading information for application.]
[Edited, 2018-03-12: Added link to guidelines for proposals]

Currently, GrimoireLab includes a tool for reporting: Manuscripts. This tool reads data from a GrimoireLab ElasticSearch database, and produces with it a PDF report with relevant metrics for a set of analyzed projects. Internally, Manuscripts uses some Python code to produce charts and CSV tables, which are integrated into a LaTeX document to produce the final PDF. Other approaches, such as producing Jupyter notebooks, will be explored too.

This idea is about adding support to Manuscripts to produce reports based on the work of the CHAOSS Community. Since Manuscripts is still a moving target, this will be also a chance to participate in the general development of the tool itself, to convert it into a generic reporting system for GrimoireLab data.

The aims of the project are as follows:

  • Writing Python code to query GrimoireLab Elastisearch databases and obtain from it the metrics relevant for the report. Possible technologies to achieve this aim include Python Pandas.
  • Writing Python code to produce suitable representation for those metrics, such as tables and charts.
  • Adapting current tools to produce reports directly from data sources, by managing the GrimoireLab toolchain. Possible solutions include adding the code to Mordred, the tool orchestrating GrimoireLab tools.

Other aims, such as producing Jupyter notebooks as a final result or an intermediate step are completely within scope.

  • Difficulty: easy/medium
  • Requirements: Python programming. Interest in software analytics. Willingness to understand GrimoireLab internals.
  • Recommended: Experience with Python interfaces to databases would be convenient, but can be learned during the project. Experience with Latex and/or Python Jupyter Notebooks would help.
  • Mentors: @jgbarah , @germonprez , @jcabot

Microtasks

For becoming familiar with the GitLab technology, it is useful if you produce an analysis (Elasticsearch indexes and dashboard) for a GitHub project (git repos and GitHub issues and pull requests). Have a look at the "Before you start" chapter in the GrimoireLab tutorial to learn how to install the needed infrastructure (ElasticSearch, Kibiter, MariaDB) and GrimoireLab itself as Python packages, or how to deploy everything as a Docker container.

Once you're familiar with producing analysis, you can exploit the information in the indexes via a Python script, preferably presented as a Python Jupyter Notebook:

  • Microtask 1: Produce a listing of the number of new committers per month, and the number of commits for each of them, as a table and as a CSV file. Use the GrimoireLab enriched index for git.

  • Microtask 2: Produce a chart showing the distribution of time-to-close (using the corresponding field in the GrimoireLab enriched index for GitHub issues) for issues already closed, and opened during the last six months.

  • Microtask 3: Produce a listing of repositories, as a table and as CSV file, with the number of commits authored, issues opened, and pull requests opened, during the last three months, ordered by the total number (commits plus issues plus pull requests).

  • Microtask 4: Perform any other analysis you may find interesting, based on GrimoireLab enriched indexes for git and GitHub repositories.

If you want, you can also:

  • Microtask 5: Produce a pull request for any of the GrimoireLab tools, and try to follow instructions until it gets accepted. Try do do something simple that you consider useful, not necessarily fix to the code: improvement of comments, documentation or testing will usually be easier to get accepted, and very useful for the project. Please, avoid just producing a random pull request just to have another microtask: the objective is not that you get one more microtask done, but that you understand how to interact with developers in the project contributing with something that could be useful).

Of course, there is no need to do all the microtasks, you only need to show that your skills are in good standing for working in this project.

Showing the work you did

If you want to show the work you did, open a GitHub repository, and upload to it:

  • A README.md file explaining what you did, and linking to the results (which will be in the same repository, see below). This will be the main file to show your skills and interest on the project, so try to make it organized and clear, in a way that we can easily understand what you did.

  • Screenshots of the dashboard(s) you produced, and configuration files you used for them (if any). Please remember removing passwords and/or auth tokens that could be in them, before uploading. In the REAME.md file, link all those files, telling about which GitHub projects you analyzed.

  • Python scripts and/or Python Jupyter notebooks you produced, with enough information (in comments and/or in the README.md file) so that we can run them if needed. Upload Python scripts ready to work assuming GrimoireLab packages are already installed. Upload Jupyter notebooks ready to be seen via the GitHub web interface.

  • Links to the pull request you did (if any), along with any comment you may have about it.

Submitting information for the application process

  • You must complete at least one micro-task related to the idea you are interested on.

  • Once you completed at least one micro-task, go to the governance repository and create a pull request to add yourself, your information, and a link to your repository with the completed micro-task(s) in the GSoC-interest.md file (see above for the contents of the repository).

  • You are welcome to include in your repository other information that could be of interest, such as open issues or pull requests submitted to the project to which you intend to contribute during GSoC, contributions to other projects, skills, and other related information.

  • You must complete these things by March 27 16:00 UTC. Make sure to also submit the information required by GSoC for applicants (i.e., project proposal), linking to it from your pull request in the GSoC-interest.md file.

  • Message to the mailing list with guidelines on proposals.

Useful documentation

Asking for help

If you need help, please use the following channels.

For issues related to GrimoireLab:

For general issues related to CHAOSS or CHAOSS metrics

@apsknight
Copy link

apsknight commented Feb 17, 2018

Hey @jgbarah

I'd like to work on this project for Google Summer of Code 2018.
Please help me getting started.

@jgbarah
Copy link
Contributor Author

jgbarah commented Feb 18, 2018

Hi, @apsknight have a look at the description of this issue, and let me know if you need further clarifications.

@vishnoitanuj
Copy link

vishnoitanuj commented Feb 19, 2018

Hi @jgbarah
I had worked on projects related metrics and was trying to gain an insight at GrimoireLab. I am not much aware of the Elasticsearch databases. So can you suggest some good resources for its training, so that i can move forward with the idea.
Also i am facing issues with understanding Mordred.

@jgbarah
Copy link
Contributor Author

jgbarah commented Feb 19, 2018

Hi @vishnoitanuj. For Elasticsearch, I've added in the description of this issue two refernces, to the most usual Python packages to access it. I would recoment going through them, and maybe some tutorial for starters on Elasticsearch.

WRT Mordred, please let me know what issues you're facing, I will try to produce documentation to help you address them, if I can.

@jsdabrowski
Copy link

Hi @jgbarah , is it suitable for you if I create my private repo with the work and share it with you and your colleagues once I finished? Sorry if the question is trivial, but I was a bit confused by saying "open a GitHub repository, and upload to it:"

Best,
Jacek

@gopal86
Copy link

gopal86 commented Feb 20, 2018

Hello @jgbarah , i am quite interested in contributing in this project . Can you guide me through ?

@jgbarah
Copy link
Contributor Author

jgbarah commented Feb 21, 2018

@jsdabrowski: the idea is that you open a public repository, and work in it. We will publish soon how to inform us about the url of the repository.

@jgbarah
Copy link
Contributor Author

jgbarah commented Feb 21, 2018

@gopal86, did you have a look at the microtasks and guidelines in the description of this issue?

@sreetamparida
Copy link

I need to ask that to work on this project for Google summer of code, I need to compete those microtask provided??

@aswanipranjal
Copy link

Hey @sreetamparida, these tasks are for you to understand the libraries and how to analyse repositories using GrimoireLab tools and ELasticsearch. You can also complete these tasks to show that you are capable and comfortable in working with the above listed technologies.
As written above:

Of course, there is no need to do all the microtasks, you only need to show that your skills are in good standing for working in this project.

@jgbarah
Copy link
Contributor Author

jgbarah commented Feb 25, 2018

@sreetamparida we need to evaluate candidates. If we have people fulfilling the microtasks, that will help us a lot to know about their skills.

@jgbarah
Copy link
Contributor Author

jgbarah commented Feb 25, 2018

@aswanipranjal thanks for your comment! Yes, it is as you say. In fact, it will not only something that will help us to decide, but also you to know about the project, and about your own skills in this area.

@apoorvkhare07
Copy link

@jgbarah How to update you about the repository ?
I have almost done the first microtask.

@jgbarah
Copy link
Contributor Author

jgbarah commented Feb 28, 2018

@apoorvkhare07 we're about to publish a message in the mailing list about that. Please, stay tuned!

@f2014169
Copy link

f2014169 commented Mar 1, 2018

@jgbarah Was the message published? , because I joined the mailing list recently.

@apoorvkhare07
Copy link

@f2014169 Not yet.

@jsdabrowski
Copy link

@jgbarah , could you please include in the mail some information related to a project proposal (what do you expect from it, and what should be included) ?

@germonprez
Copy link

@jsdabrowski @apoorvkhare07 @f2014169 you can see the message at:

https://lists.linuxfoundation.org/pipermail/oss-health-metrics/2018-February/000517.html

@Deepayan-Ghosh
Copy link

@jgbarah I am interested in this project. I have installed the necessary tools and have gone through the tutorials and also ran some code. However I had one question in my mind, which may be trivial. The question is:
Are both raw and enriched indices the same for elasticsearch? That is, whether both of them appear as just indexes for elasticsearch, and whether the enriched index is something that is made by ELK exclusively just to facilitate making dashboards?
Isn't there any programmatic way of creating and uploading the enriched index to elasticsearch, instead of running the p2o.py script?

Thank you

@gopal86
Copy link

gopal86 commented Mar 3, 2018

hi @jgbarah , @germonprez . I have completed the microtask-1 . can you guide as to what to do next ?

@f2014169
Copy link

f2014169 commented Mar 3, 2018

@gopal86 please visit https://lists.linuxfoundation.org/pipermail/oss-health-metrics/2018-February/000517.html

@f2014169
Copy link

f2014169 commented Mar 4, 2018

@jgbarah do you have any specific format for proposal??

@jgbarah
Copy link
Contributor Author

jgbarah commented Mar 5, 2018

@jgbarah do you have any specific format for proposal??

Not for now. But let me know if the elements that should be included in it are not clear.

@jgbarah
Copy link
Contributor Author

jgbarah commented Mar 5, 2018

Are both raw and enriched indices the same for elasticsearch? That is, whether both of them appear as just indexes for elasticsearch, and whether the enriched index is something that is made by ELK exclusively just to facilitate making dashboards?

The raw index is intended to store data as similar as possible to the original data source. Raw indexes are usually rich JSON documents, with all the details obtained from the service API or similar.

The enriched index is produced from the raw index, for being visualized with Kibiter/Kibana. It is usually a flat JSON document (because nested documents have some trouble with Kibana), which is in part a summary of the documents in the raw index (because only those properties relevant for visualizing are included), and the result of some computing for some parameters (eg, time to close a ticket, which is not directly available from the API, but has to be computed from the time the ticket was opened, to the time it was closed).

Isn't there any programmatic way of creating and uploading the enriched index to elasticsearch, instead of running the p2o.py script?

Yes. p2o.py is just a Python script with uses the grimoire_elk module to fetch data (using Perceval), enrich it, and upload it to Elasticsearch.

@jsdabrowski
Copy link

@jgbarah what would you like to be Included in the proposal? Would you like a candidate to propose how to approach the project? If so, then more detail are needed. Otherwise, cover letter is enough?

@kmn5409
Copy link
Member

kmn5409 commented Mar 6, 2018

Just wondering would you be able to use Linux Bash Shell on Windows 10 for the things that will be installed for example Elasticsearch. Or would you use something like a virtual machine like Oracle VM?

@jgbarah
Copy link
Contributor Author

jgbarah commented Mar 7, 2018

@jsdabrowski the proposal could be as much detailed as you feel convenient. If you have doubts on the scope, or on what is asked for, that's my fault: let me know what you're missing, and we'll complete the description of this issue as much as needed. If you're doubting between different approaches, that's no problem: either explain the alternatives you see, or just select one and expose why you think it is better. That does not mean we're going to decide to go that way, but it is going to tell us about how you deal with selecting choices...

But again, if you feel you need more info, let us know which info you are missing, please.

@jgbarah
Copy link
Contributor Author

jgbarah commented Mar 7, 2018

@kmn5409 I'm not sure what do you mean. If that is if you can run Elasticsearch from a bash (or a powershell) in Windows, I assume you can, although I'm not familiar with Windows, sorry. If that is something else, please let me know.

However, for clarifying, almost all the development for GrimoireLab is done in Linux-based systems. That means that using Linux for development is certainly more friendly, and we can provide much more support.

@jgbarah
Copy link
Contributor Author

jgbarah commented Mar 8, 2018

@aswanipranjal said in chaoss/grimoirelab-tutorial#19:

I had some questions about the project Reporting of CHAOSS metrics:
To extend the functionality of Manuscripts, we will need to add code to calculate the Metrics listed down at https://github.com/chaoss/metrics, right? If so, how do you suggest that the user demographics be collected and monitored?

We may need too add code, or to reuse some code and write the rest from scratch. We welcome your ideas and proposals on how to tackle that, based on your understanding of the code.

With respect to specific metrics, please have in mind that not all of them can be retrieved automatically from software development repositories. Some of them could be, for example, collected manually, and written into some properly formatted YAML file, which Manuscripts would read to produce the relevant metrics. But again, we're open to your ideas.

In the 3rd Microtask for the project, it is said that: Produce a listing of repositories, as a table and as CSV file, with the number of commits authored, issues opened, and pull requests opened, during the last three months, ordered by the total number (commits plus issues plus pull requests). Does that mean all the public repositories created in the last 3 months? Or does that mean that all the repositories should be listed according to the number of commits authored + pull requests made + issues opened made in the last 3 months? If the answer is the latter one: then how many repositories should be listed out?

It is the latter, and the number is all of them. Please remember that the idea is that this should run on GrimoireLab indexes, which means the real answer is "all repositories in the indexes considered". In other words, the code should just look at the indexes, find all the repositories in them, and produce numbers with their activity during the last three months (including those with zero activity).

Is it more clear now?

@kmn5409
Copy link
Member

kmn5409 commented Mar 19, 2018

I'm not sure but when I try to create the Kibana dashboard I'm getting the error
$ kidash --elastic_url-enrich http://localhost:9200 --import /tmp/git-dashboard.json 2018-03-19 09:29:59,907 Panel not found (not in directory, no panels module): /tmp/git-dashboard.json 2018-03-19 09:29:59,907 Wrong file format (can't find 'dashboard' field): /tmp/git-dashboard.json
Now I might be putting the tmp folder in the wrong place, where is it supposed to go exactly, I have placed it in a few places in my (perceval) virtual environment. Also I tried looking online and I saw that I could just have the .json file in the same folder. I used this command
$ kidash --elastic_url-enrich http://localhost:9200 \ --import git-dashboard.json usage: usage: kidash [options] kidash: error: unrecognized arguments: --import git-dashboard.json
But then I get that error.
The directory I'm in looks like
$ ls git-dashboard.json perceval_github_1.py tmp token.txt

@valeriocos
Copy link
Member

Hi @kmn5409,
A similar problem has been reported and successfully addressed in #71 (look for /tmp/git-dashboard.json in the comments).
Let me know if this helps you to solve your problem.

@kmn5409
Copy link
Member

kmn5409 commented Mar 19, 2018

@valeriocos Yeah so this was what I had used to try and fix my problem but I still kept getting other issues

@valeriocos
Copy link
Member

@kmn5409, can you tell more about the other issues (or the other issues are the ones above?)

@kmn5409
Copy link
Member

kmn5409 commented Mar 20, 2018

So @valeriocos I'm trying to follow the tutorial for https://grimoirelab.gitbooks.io/tutorial/grimoireelk/a-simple-dashboard.html and when I try to create the dashboard I get the issue
$ kidash --elastic_url-enrich http://localhost:9200 --import /tmp/git-dashboard.json 2018-03-19 09:29:59,907 Panel not found (not in directory, no panels module): /tmp/git-dashboard.json 2018-03-19 09:29:59,907 Wrong file format (can't find 'dashboard' field): /tmp/git-dashboard.json

@jgbarah
Copy link
Contributor Author

jgbarah commented Mar 20, 2018

@Deepayan-Ghosh said:

This is related to the proposal. The goal of the project is to add support to manuscripts to produce reports of any repositories based on the metrics in https://github.com/chaoss/metrics. For example, now the pdf report that is generated has sections on: "Overview","Activity","Community" and "Process", however we want to add support for the metrics provided in https://github.com/chaoss/metrics like Diversity-Inclusion, Risk, etc and for each of these categories, analyse the indices, obtain relevant information and produce representations of metrics like "Contributor demographics" for Diversity-Inclusion category, and so on. Is my understanding correct??

In short, yes. But details could be different. For example, we could use a combination of scripts and Python notebooks for producing the result. But first step will be doing that with the current Manuscripts machinery, the rest will depend on progress and ideas.

@jgbarah
Copy link
Contributor Author

jgbarah commented Mar 20, 2018

@kmn5409 said:

So @valeriocos I'm trying to follow the tutorial for https://grimoirelab.gitbooks.io/tutorial/grimoireelk/a-simple-dashboard.html and when I try to create the dashboard I get the issue
$ kidash --elastic_url-enrich http://localhost:9200 --import /tmp/git-dashboard.json 2018-03-19 09:29:59,907 Panel not found (not in directory, no panels module): /tmp/git-dashboard.json 2018-03-19 09:29:59,907 Wrong file format (can't find 'dashboard' field): /tmp/git-dashboard.json

Which file are you using for /tmp/git-dashboard.json? Becase it seems the file is not in the right format.

@jgbarah
Copy link
Contributor Author

jgbarah commented Mar 20, 2018

@Deepayan-Ghosh said:

I was following this tutorial: https://grimoirelab.gitbooks.io/tutorial/mordred/a-grimoirelab-dashboard-in-one-step.html . Downloaded the mordred-grimoirelab.cfg file, menu.yaml, and project.json files. I also cloned the directory containing the panels but when I run mordred -c mordred-grimoirelab.cfg I get the following error:
...

The only thing I see different from my setup is that you're using Kibana 6.2.x, and the latest release I've tried is Kibana 6.1.x. Would you mind trying with Kibana (and Elasticsearch) 6.1.x? Meanwhile, I will try with 6.2.x, to see if we have some trouble with it.

Also if I change post to get at https://github.com/chaoss/grimoirelab-mordred/blob/master/mordred/task_panels.py#L361 then I do not get the error.

Maybe this means something changed in ES 6.1x to 6.2.x... Not sure about that. I will check. BTW, please ensure that you have the latest GrimoireLab pip packages, since those available one week ago were known not to work with ES 6.x. (If you don't mind, tell me which version of the grimoire-mordred package are you using).

@valeriocos
Copy link
Member

@kmn5409 I'm not able to replicate your error. Below, the steps I did:

  • Start your ES (let's suppose that your ES is running on http://localhost:9200)
  • Run these commands where your p2o.py is located (in my case /home/GrimoireELK/utils/p2o.py):
    p2o.py --enrich --index git_raw --index-enrich git -e http://localhost:9200 --no_inc --debug git https://github.com/grimoirelab/perceval.git
    p2o.py --enrich --index git_raw --index-enrich git -e http://localhost:9200 --no_inc --debug git https://github.com/grimoirelab/GrimoireELK.git
  • Check that there is something in your indexes:
    http://localhost:9200/git_raw?pretty=true
    http://localhost:9200/git?pretty=true
  • Click on https://grimoirelab.gitbooks.io/tutorial/grimoireelk/dashboards/git-dashboard.json, right click, save as git-dashboard.json wherever you want (for instance /tmp/)
  • Run this command where kidash is located (in my case /home/grimoirelab-kidash/bin/kidash):
    kidash --elastic_url-enrich http://localhost:9200 --import /tmp/git-dashboard.json

Let me know if this helps to solve your issue.

@Deepayan-Ghosh
Copy link

@jgbarah The same modred config file works and does not give any error, when I used Kibana (and elasticsearch) 6.1.0 and 6.1.3 and 6.1.4 versions. However there is an error with the default index pattern that I previously mentioned here: #70 (comment) and hence I moved to using lastest 6.2.2 versions.

On the other hand, I had mordred version 0.1.15 and I upgraded and reinstalled using pip to get version 0.1.16. However, I see in the github repo that version.py of mordred was just updated to 0.1.19. Trying to use the mordred config file using mordred 0.1.16 still gives error:

Kibiter/Kibana: version found is 6.2.2
Kibiter/Kibana: configured!
Dashboard panels, visualizations: uploading...
Dashboard panels, visualizations: uploaded!
Dashboard menu: uploading...
Dashboard menu: uploaded!
Collection for git: starting...
Collection for git: finished after 00:00:04 hours
Enrichment for git: starting...
Elasticsearch aliases for git: creating...
2018-03-21 03:19:26,568 - mordred.task_manager - ERROR - Exception in Task Manager 400 Client Error: Bad Request for url: http://127.0.0.1:9200/_aliases
Exception in thread git:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/deepayan/Grimoirelab/grimoirelab/lib/python3.6/site-packages/mordred/task_manager.py", line 92, in run
    task.execute()
  File "/home/deepayan/Grimoirelab/grimoirelab/lib/python3.6/site-packages/mordred/task_enrich.py", line 253, in execute
    self.__enrich_items()
  File "/home/deepayan/Grimoirelab/grimoirelab/lib/python3.6/site-packages/mordred/task_enrich.py", line 145, in __enrich_items
    task_aliases.execute()
  File "/home/deepayan/Grimoirelab/grimoirelab/lib/python3.6/site-packages/mordred/task_panels.py", line 400, in execute
    self.__create_aliases()
  File "/home/deepayan/Grimoirelab/grimoirelab/lib/python3.6/site-packages/mordred/task_panels.py", line 392, in __create_aliases
    self.__create_alias(es_enrich_url, index_enrich, alias)
  File "/home/deepayan/Grimoirelab/grimoirelab/lib/python3.6/site-packages/mordred/task_panels.py", line 363, in __create_alias
    res.raise_for_status()
  File "/home/deepayan/Grimoirelab/grimoirelab/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://127.0.0.1:9200/_aliases

Also I successfully created report using manuscripts for git as data_source. However, if I use github_issues as data_source then I get this error, which on comparison with git.py in manuscripts I see that https://github.com/chaoss/grimoirelab-manuscripts/blob/master/manuscripts/metrics/github_issues.py#L36 does not have anything mentioned in the list:

Traceback (most recent call last):
  File "/home/deepayan/grimoirelab/grimoirelab/bin/manuscripts", line 130, in <module>
    report.create()
  File "/home/deepayan/grimoirelab/grimoirelab/lib/python3.6/site-packages/manuscripts/report.py", line 711, in create
    self.create_data_figs()
  File "/home/deepayan/grimoirelab/grimoirelab/lib/python3.6/site-packages/manuscripts/report.py", line 603, in create_data_figs
    self.sections()[section]()
  File "/home/deepayan/grimoirelab/grimoirelab/lib/python3.6/site-packages/manuscripts/report.py", line 286, in sec_overview
    author = self.config['overview']['author_metrics'][0]
IndexError: list index out of range

@kmn5409
Copy link
Member

kmn5409 commented Mar 21, 2018

@valeriocos I don't know if I'm saving the git-dashboard.json file in the wrong place, when you say the /tmp/ folder where is that exactly. I get the same issue as before when I run the command
kidash --elastic_url-enrich http://localhost:9200 --import /tmp/git-dashboard.json 2018-03-21 08:22:32,426 Panel not found (not in directory, no panels module): /tmp/git-dashboard.json 2018-03-21 08:22:32,426 Wrong file format (can't find 'dashboard' field): /tmp/git-dashboard.json
After following your steps of using p2o.py and then ensuring that something is in the indexes, those work fine

@kmn5409
Copy link
Member

kmn5409 commented Mar 21, 2018

Also I would like to know about how Microtask #5, so when it comes to providing things like documentation, it says that we can do it on any of the GrimoireLab tools so would it also include repositories like chaoss/grimoirelab-perceval and even the repository we're in now chaoss/grimoirelab?

@valeriocos
Copy link
Member

valeriocos commented Mar 21, 2018

@kmn5409 , open a shell and write cd /tmp/ (you will be in the /tmp/ folder). In a similar way you could store the git-dashboard.json in another directory (for instance with cd ~/, you will be in your home directory).
Can you try to use the .json file in git-dashboard.json.tar.gz instead of the one you are using?

@kmn5409
Copy link
Member

kmn5409 commented Mar 21, 2018

Okay thank you for the responses @valeriocos so I saved it in the tmp folder now and when I run the command I now have this error
kidash --elastic_url-enrich http://localhost:9200 --import /tmp/git-dashboard.json Traceback (most recent call last): File "/home/keanu/venvs/perceval/bin/kidash", line 77, in <module> ARGS.data_sources, ARGS.add_vis_studies) File "/home/keanu/venvs/perceval/lib/python3.5/site-packages/kidash/kidash.py", line 703, in import_dashboard feed_dashboard(dashboard, elastic_url, es_index, data_sources, add_vis_studies) File "/home/keanu/venvs/perceval/lib/python3.5/site-packages/kidash/kidash.py", line 720, in feed_dashboard dashboard['dashboard']['value'], data_sources, add_vis_studies) File "/home/keanu/venvs/perceval/lib/python3.5/site-packages/kidash/kidash.py", line 241, in import_item_json res.raise_for_status() File "/home/keanu/venvs/perceval/lib/python3.5/site-packages/requests/models.py", line 935, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://127.0.0.1:9200/.kibana/doc/dashboard:Git

@valeriocos
Copy link
Member

You're welcome @kmn5409 , do you have Kibana installed?

@kmn5409
Copy link
Member

kmn5409 commented Mar 22, 2018

screenshot from 2018-03-21 20-29-15
To the best of my knowledge yes, is this how you check? @valeriocos

@gopal86
Copy link

gopal86 commented Mar 23, 2018

I was running this command as given in the tutorial kidash --elastic_url-enrich http://localhost:9200 --import /tmp/git-dashboard.json. I am getting the following error :-

Traceback (most recent call last): File "/home/vallabh/venvs/grimoirelab/bin/kidash", line 33, in <module> from kidash.kidash import import_dashboard, export_dashboard, list_dashboards File "/home/vallabh/venvs/grimoirelab/bin/kidash.py", line 33, in <module> from grimoire_elk.panels import import_dashboard, export_dashboard, list_dashboards ImportError: No module named 'grimoire_elk.panels'

Can anyone help me out ?

@valeriocos
Copy link
Member

valeriocos commented Mar 23, 2018

Hi @gopal86, have you imported/installed the sigils/panels package?

@gopal86
Copy link

gopal86 commented Mar 23, 2018

yes @valeriocos , i had installed it . it also shows Successfully installed kidash-0.4.3 .

@valeriocos
Copy link
Member

It's possible that you are using an old version of grimoire elk. Please @gopal86, try with the latest version.

@valeriocos
Copy link
Member

@kmn5409, I'm not able to replicate your issue. Do you have the latest version of kidash, ELK?
Is the version of your ElasticSearch aligned with the one of Kibana?

@kmn5409
Copy link
Member

kmn5409 commented Mar 26, 2018

@valeriocos I saw that it might have been the memory on my computer was filled up so I had to make room on it, I assume that was the issue because when I tried to upload things to elasticsearch it was not working, so I think I got it out, and also to answer your question I use docker to get my instance of kibana and elasticsearch.

@LShivaPrabhakar
Copy link

Hey @jgbarah

I'd like to work on this project for Google Summer of Code 2018.
Please help me getting started.

@germonprez
Copy link

@LShivaPrabhakar

Thanks for interest. We do have tasks that we ask potential students to complete. They can be found here:

https://lists.linuxfoundation.org/pipermail/oss-health-metrics/2018-February/000517.html

You should also note that the student application deadline is: March 27 16:00 UTC.

@jgbarah
Copy link
Contributor Author

jgbarah commented Sep 11, 2018

GSoC is well over, so I'm closing this issue.

@jgbarah jgbarah closed this as completed Sep 11, 2018
@urstrulykkr
Copy link

Hi@jgbarah
I would like to take up idea #2 as my GSoC-19 project in the link provided
https://wiki.linuxfoundation.org/chaoss/gsoc-ideas
Also, looking to patch few bugs being reported in the issue section of the grimoirelab repo
https://github.com/chaoss/grimoirelab/issues

what would you say ?

@urstrulykkr
Copy link

@jgbarah needed your guidance regarding GSoC-19 project contribution.

@jgbarah
Copy link
Contributor Author

jgbarah commented Jan 29, 2019

@urstrulykkr sorry for being so late with this. In case you're still interested: this issue was for GSoC'18. For '19, we're still in the stage of proposing our project for participate in GSoC'19. We will open issues with the new tasks in due time. Thanks a lot for your interest!

@urstrulykkr
Copy link

urstrulykkr commented Jan 30, 2019 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests