Issue running areas of code study

[jsmanrique]

I am getting the following error running an areas of code analysis:

Executing for git the studies ['enrich_demography', 'enrich_areas_of_code']
/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/pandas/core/ops.py:816: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
 result = getattr(x, name)(y)
2018-05-11 07:46:03,905 - grimoire_elk.elk - ERROR - Problem executing study <bound method GitEnrich.enrich_areas_of_code of <grimoire_elk.enriched.git.GitEnrich object at 0x7f0f6ccf9c18>>
Traceback (most recent call last):
 File "/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/grimoire_elk/elk.py", line 460, in do_studies
   study(enrich_backend, no_incremental)
 File "/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/grimoire_elk/enriched/git.py", line 787, in enrich_areas_of_code
   areas_of_code(git_enrich=enrich_backend, in_conn=in_conn, out_conn=out_conn)
 File "/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/grimoire_elk/enriched/study_ceres_aoc.py", line 194, in areas_of_code
   ndocs = aoc.analyze()
 File "/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/grimoire_elk/enriched/ceres_base.py", line 93, in analyze
   process_results = self.process(item_block)
 File "/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/grimoire_elk/enriched/study_ceres_aoc.py", line 158, in process
   events_df = data_filtered.filter_(["filepath"], "-")
 File "/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/cereslib/dfutils/filter.py", line 71, in filter_
   self.data = self.data[self.data[column] != value]
 File "/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/pandas/core/ops.py", line 879, in wrapper
   res = na_op(values, other)
 File "/home/jsmanrique/grimoirelab/venv/lib/python3.5/site-packages/pandas/core/ops.py", line 818, in na_op
   raise TypeError("invalid type comparison")
TypeError: invalid type comparison

Config files:

• projects.json.txt

[alpgarcia]

It seems it is working in my local env, with ES 5.1. Next time you find this error it would be useful to look for log lines like:
https://github.com/alpgarcia/GrimoireELK/blob/112bfb45c3b62bdffcf2451e08fe6d3e9c61c4f3/grimoire_elk/enriched/study_ceres_aoc.py#L148
https://github.com/alpgarcia/GrimoireELK/blob/112bfb45c3b62bdffcf2451e08fe6d3e9c61c4f3/grimoire_elk/enriched/study_ceres_aoc.py#L154

2018-05-11 16:16:05,217 New commits: 100
2018-05-11 16:16:05,684 New events: 131

to know if we are having problems reading items.

[jsmanrique]

Thanks @alpgarcia. Have you tried with ES 6.2.1 instead?

[alpgarcia]

Nope, because we are not supporting ES 6 yet, but we need to add this to the checks asap.

[alpgarcia]

Taking this issue to repeat the same procedure in ES 6.2.1.

@jsmanrique I guess you were using Kibiter, right?

[jsmanrique]

Yes, I was

[alpgarcia]

I've tried again using the following images (latest images currently available):

  image: bitergia/elasticsearch:6.1.0
  image: bitergia/kibiter:6.1.0-optimized

And running p2o against your RAW git index. I didn't get any error after processing 17.5k items, so we can be pretty sure the code works also here.

At this point it seems an error inherited from your environment. @jsmanrique, could you please paste here the result of executing pip3 freeze inside your execution env?

[jsmanrique]

beautifulsoup4==4.6.0
brewer2mpl==1.4.1
cereslib==0.1.3
certifi==2018.4.16
chardet==3.0.4
cheroot==5.8.3
CherryPy==11.0.0
click==6.7
colorama==0.3.9
cycler==0.10.0
Django==2.0.5
dulwich==0.18.6
elasticsearch==6.2.0
elasticsearch-dsl==6.1.0
feedparser==5.2.1
grimoire-elk==0.30.37
grimoire-mordred==0.1.26
grimoire-reports==0.1.5
grimoirelab-panels==0.0.12
grimoirelab-toolkit==0.1.7
httpretty==0.8.14
idna==2.6
indexwarrior==0.2.1
ipaddress==1.0.22
Jinja2==2.10
kidash==0.4.4
kingarthur==0.1.8
kiwisolver==1.0.1
manuscripts==0.2.10
MarkupSafe==1.0
matplotlib==2.2.2
numpy==1.14.3
pandas==0.23.0
perceval==0.10.3
perceval-mozilla==0.1.13
perceval-opnfv==0.1.8
perceval-puppet==0.1.7
pkg-resources==0.0.0
portend==2.2
prettyplotlib==0.1.7
prettytable==0.7.2
PyMySQL==0.8.1
pyparsing==2.2.0
python-dateutil==2.7.3
pytz==2018.4
PyYAML==3.12
redis==2.10.6
requests==2.18.4
rq==0.10.0
scipy==1.1.0
six==1.11.0
sortinghat==0.6.3
SQLAlchemy==1.2.7
tempora==1.11
urllib3==1.22

[jsmanrique]

My mordred.cfg file:

[general]
short_name = Project_Name
update = false
min_update_delay = 10
debug = true
logs_dir = /tmp/logs

[projects]
projects_file = projects.json

[es_collection]
url = http://localhost:9200
user =
password =

[es_enrichment]
url = http://127.0.0.1:9200
user =
password =
autorefresh = true

[sortinghat]
host = localhost
user = root
password = *******************
database = shdb
load_orgs = true
orgs_file = orgs.json
unaffiliated_group = Unknown
affiliate = true
autoprofile = [github,git]
matching = [email-name]
sleep_for = 0
#bots_names = []

[panels]
kibiter_time_from= "now-90d"
kibiter_default_index= "git"

[phases]
collection = true
identities = true
enrichment = true
panels = true

[git]
raw_index = git-gathered-raw
enriched_index = git-gathered
studies = [enrich_demography, enrich_areas_of_code]

[github]
raw_index = github-gathered-raw
enriched_index = github-gathered
api-token = **************************************************
sleep-for-rate = true
sleep-time = 300
no-archive = true

[*pipermail]
raw_index = mls-gathered-raw
enriched_index = mls-gathered

[*meetup]
raw_index = meetup-gathered-raw
enriched_index = meetup-gathered
api-token = *****************************************
sleep-for-rate = true
no-archive = true

[*stackexchange]
raw_index = stackexchange-gathered-raw
enriched_index = stackexchange-gathered
api-token = *************************************

[alpgarcia]

After updating pandas to 0.23 the code is still working properly. I think this issue should be in https://github.com/chaoss/grimoirelab-sirmordred as I am unable to reproduce it using gelk.

@acs what do you think?

[acs]

After updating pandas to 0.23 the code is still working properly. I think this issue should be in https://github.com/chaoss/grimoirelab-sirmordred as I am unable to reproduce it using gelk.

@acs what do you think?

Hmmm, it could be an option. But if we can not reproduce it in our supported envs, I will close the ticket and recommend to use the supported envs.

[alpgarcia]

Moving it to mordred as I understand @jsmanrique is using a supported mordred version.

[acs]

Hmmm, the supported version must be of the full platform: grimoirelab + elasticsearch + kibana. Basically, we are supporting now ELK 5.1.1 and ELK 6.1.0. In any case, let's continue with this issue in mordred. Thanks @alpgarcia !