db uri

[jlousada315]

Hi !

Can you please clarify, where I can get the db uri ? I am not able to cache the raw data .

Thank you in advance !

[afrittoli]

Hi - you can get the raw data using the default_db_uri https://github.com/mtreinish/ciml/blob/master/ciml/gather_results.py#L37
More details: https://docs.openstack.org/infra/system-config/logstash.html#subunit2sql

We might want to add this info on the main README.md ^_^

[jlousada315]

Hi , thank you for answering in such short notice !

In the command line, when I write, for example: ciml-cache-data --build-name tempest-full --db-uri mysql+pymysql://query:query@logstash.openstack.org/

I get an error message saying: pymysql.err.InternalError: (1046, 'No database selected')

Any idea why this might be ?

Also, from the README file, I think it is not very straightforward to get to know how to use the API, where can I find all the build names possible ?

[jlousada315]

Hi again !

I managed to connect the DB:

ciml-cache-data --build-name tempest-full --db-uri mysql+pymysql://query:query@logstash.openstack.org/subunit2sql

Obtained 2592 runs named tempest-full from the DB
Check-point 0/2592, saved 0, skipped 0

But I get the error:
TypeError: cannot unpack non-iterable NoneType object.

[afrittoli]

Thank you for your issue report. It looks like the case of local storage (as opposed to s3) is not working, a change in the signature of _get_data_handlers broke it.

I'll leave this open as a documentation issue, I filed #44 for the code.

[jlousada315]

Thank you, that issue is resolved, but now I get the following exception:

requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

when I try to acces log stash.openstack.org/subunit2sql on my browser , it states No URL found.

[jlousada315]

Hi ! Sorry to bother again, but I keep trying and can't figure out what's wrong. I'm using the data to develop a similar work for my master thesis and such quality data would be so helpful.

thank you once more, for trying to figure out what's wrong

[jlousada315]

Now the error is:

Obtained 2559 runs named tempest-full from the DB
Check-point 0/2559, saved 0, skipped 0
b75eb96d-15ce-4591-9de6-96c003b84520: dstat found in cache
b75eb96d-15ce-4591-9de6-96c003b84520: dstat found in cache
b75eb96d-15ce-4591-9de6-96c003b84520: metadata found in cache
0[b75eb96d-15ce-4591-9de6-96c003b84520]: Data found
68e0601b-63da-4635-94b9-8256af0affb0: dstat found in cache
Traceback (most recent call last):
File "/usr/local/bin/ciml-cache-data", line 10, in
sys.exit(cache_data())
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/Users/joaolousada/Documents/5ºAno/Master-Thesis/ciml-master/ciml/gather_results.py", line 730, in cache_data
s3_url)
File "/Users/joaolousada/Documents/5ºAno/Master-Thesis/ciml-master/ciml/gather_results.py", line 742, in cache_data_function
runs, build_name, limit, '1s', db_uri, data_path=data_path, s3=s3)
File "/Users/joaolousada/Documents/5ºAno/Master-Thesis/ciml-master/ciml/gather_results.py", line 421, in gather_and_cache_results_for_runs
data_path=data_path, s3=s3)
File "/Users/joaolousada/Documents/5ºAno/Master-Thesis/ciml-master/ciml/gather_results.py", line 332, in _get_data_for_run
use_http=use_remote, data_path=data_path, s3=s3)
File "/Users/joaolousada/Documents/5ºAno/Master-Thesis/ciml-master/ciml/gather_results.py", line 177, in _get_dstat_file
return _parse_dstat_file(f, sample_interval, skiprows)
File "/Users/joaolousada/Documents/5ºAno/Master-Thesis/ciml-master/ciml/gather_results.py", line 110, in _parse_dstat_file
out = pd.read_csv(input_io, skiprows=skiprows).set_index('time')
File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 685, in parser_f
return _read(filepath_or_buffer, kwds)
File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 457, in _read
parser = TextFileReader(fp_or_buf, **kwds)
File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 895, in init
self._make_engine(self.engine)
File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 1135, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 1917, in init
self._reader = parsers.TextReader(src, **kwds)
File "pandas/_libs/parsers.pyx", line 545, in pandas._libs.parsers.TextReader.cinit
pandas.errors.EmptyDataError: No columns to parse from file

[afrittoli]

Thank you, that issue is resolved, but now I get the following exception:

requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

when I try to acces log stash.openstack.org/subunit2sql on my browser , it states No URL found.

logstash.openstack.org/subunit2sql is the URL of a mysql database, so you need to use an SQL client if you want to connect directly to it. See more details on https://docs.openstack.org/infra/system-config/logstash.html#subunit2sql .

The subunit2sql project defines the DB schema and also exposes a python API to access the data in the DB.

[afrittoli]

EmptyDataError

I've not encountered the EmptyDataError before.
According to Pandas that might mean no data in the CSV file, so I wonder if you hit a case of an empty dstat file. I think adding error handling for this case should be enough to let you continue.

I filed #46 and fixed it in #47. Let me know if this solves your issue.

[afrittoli]

@johnnylousas Thank you for contributing to our project, I hope you find it useful for your work.
Should you encounter further issues, would you mind tracking them in separate github issues?
I would like to keep this one as a documentation issue.

[jlousada315]

Ok, thank you very much I will address further issues separately.

[afrittoli]

I'll keep this open for now as we need to add some docs to the README