KeyError on pipeline upload

[joaquinvanschoren]

Simple pipeline experiment:

from sklearn import pipeline, ensemble, preprocessing
from openml import tasks,runs, datasets
task = tasks.get_task(59)
pipe = pipeline.Pipeline(steps=[
            ('Imputer', sklearn.preprocessing.Imputer(strategy='median')),
            ('OneHotEncoder', sklearn.preprocessing.OneHotEncoder(sparse=False, handle_unknown='ignore')),
            ('Classifier', sklearn.ensemble.RandomForestClassifier())
           ])
run = runs.run_task(task, pipe)
response = run.publish()

Error:

KeyError                                  Traceback (most recent call last)
<ipython-input-7-bbc4e8d1aa68> in <module>()
      9 run = runs.run_task(task, pipe)
     10 #vars(run)
---> 11 response = run.publish()

/Users/joa/anaconda/lib/python3.5/site-packages/openml-0.3.0-py3.5.egg/openml/runs/run.py in publish(self)
    137 
    138         predictions = arff.dumps(self._generate_arff_dict())
--> 139         description_xml = self._create_description_xml()
    140 
    141         file_elements = {'predictions': ("predictions.arff", predictions),

/Users/joa/anaconda/lib/python3.5/site-packages/openml-0.3.0-py3.5.egg/openml/runs/run.py in _create_description_xml(self)
    164         downloaded_flow = openml.flows.get_flow(self.flow_id)
    165 
--> 166         openml_param_settings = _parse_parameters(self.model, downloaded_flow)
    167 
    168         # as a tag, it must be of the form ([a-zA-Z0-9_\-\.])+

/Users/joa/anaconda/lib/python3.5/site-packages/openml-0.3.0-py3.5.egg/openml/runs/run.py in _parse_parameters(model, flow)
    200         if isinstance(python_param_settings[param], BaseEstimator):
    201             # extract parameters of the subflow individually
--> 202             subflow = flow.components[param]
    203             openml_param_settings += _parse_parameters(python_param_settings[param], subflow)
    204 

KeyError: 'Imputer'

Am I doing something wrong?

[mfeurer]

Thanks for letting us know. Which branch is this on? We will then incorporate this to the unit tests.

[joaquinvanschoren]

This is for branch feature/upload-run.

On Wed, Nov 23, 2016 at 9:33 AM Matthias Feurer notifications@github.com
wrote:

Thanks for letting us know. Which branch is this on? We will then
incorporate this to the unit tests.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#187 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABpQV3igc7i-6junbQKq_Sg3wTyoq_wLks5rA_pHgaJpZM4K6FJJ
.

[mfeurer]

Okay, that branch is still under development. There are no guarantees that it works yet.

[joaquinvanschoren]

The exact same errors happens on the master branch.
Is this way of defining a pipeline incorrect?

On Wed, Nov 23, 2016 at 9:55 AM Matthias Feurer notifications@github.com
wrote:

Okay, that branch is still under development. There are no guarantees that
it works yet.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#187 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABpQV3V1R_Ee7IppTXcO_957nV1lyZnlks5rA_-VgaJpZM4K6FJJ
.

[mfeurer]

The pipeline looks correct. On the master branch it won't work, because the master branch can't handle pipelines yet.

[joaquinvanschoren]

Sorry, the same error pops up on develop and feature/upload-flow as well.
Anything else you want me to try?

On Wed, Nov 23, 2016 at 10:38 AM Matthias Feurer notifications@github.com
wrote:

The pipeline looks correct. On the master branch it won't work, because
the master branch can't handle pipelines yet.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#187 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABpQV8cmqCJvJx-S9S-ifUTwWLbQmDoHks5rBAmMgaJpZM4K6FJJ
.

[mfeurer]

Hm, thanks, but that's all the branches we have at the moment. It seems like this feature doesn't work yet.

[joaquinvanschoren]

Correction, I forgot to reinstall after switching branches. New information:

Feature/run-upload branch: KeyError as shown above
Develop branch: TypeError: run_task fails while checking whether the flow exists

Users/joa/anaconda/lib/python3.5/site-packages/openml-0.3.0-py3.5.egg/openml/runs/functions.py in run_task(task, model)
     42 
     43     flow = OpenMLFlow(model=model)
---> 44     flow_id = flow._ensure_flow_exists()
     45     if (flow_id < 0):
     46         print("No flow")

/Users/joa/anaconda/lib/python3.5/site-packages/openml-0.3.0-py3.5.egg/openml/flows/flow.py in _ensure_flow_exists(self)
    110 
    111         if int(flow_id) == -1:
--> 112             return_code, response_xml = self.publish()
    113 
    114             response_dict = xmltodict.parse(response_xml)

TypeError: 'OpenMLFlow' object is not iterable

master and Feature/flow-upload branch: OpenMLServerError (Invalid XML):

/Users/joa/anaconda/lib/python3.5/site-packages/openml-0.3.0-py3.5.egg/openml/_api_calls.py in _perform_api_call(call, file_dictionary, file_elements, add_authentication)
     42     if file_dictionary is not None or file_elements is not None:
     43         return _read_url_files(url, file_dictionary=file_dictionary,
---> 44                                file_elements=file_elements)
     45     return _read_url(url)
     46 

/Users/joa/anaconda/lib/python3.5/site-packages/openml-0.3.0-py3.5.egg/openml/_api_calls.py in _read_url_files(url, file_dictionary, file_elements)
     76     response = requests.post(url, data=data, files=file_elements)
     77     if response.status_code != 200:
---> 78         raise OpenMLServerError(response.text)
     79     if 'Content-Encoding' not in response.headers or \
     80             response.headers['Content-Encoding'] != 'gzip':

OpenMLServerError: OpenML Server error: <oml:error xmlns:oml="http://openml.org/openml">
	<oml:code>202</oml:code>
	<oml:message>Could not validate run xml by xsd</oml:message>
		<oml:additional_information>XML does not correspond to XSD schema. </oml:additional_information>
	</oml:error>

Is the TypeError something that is fixed in upload-run? Hence, the main problem is the KeyError?

[mfeurer]

It's definitively fixed in one of the branches. The issue is that something goes wrong when creating the parameters for the run upload. The key error is the symptom that we can observe.

[joaquinvanschoren]

Thanks. I'll take a look and see if I can find the problem.
Btw, it seems like all uploaded pipelines are called 'sklearn.pipeline.Pipeline'. Do you have a plan to generate more readable names? (I saw something about that in a PR I think)?

[mfeurer]

I'm not sure if it's a good investment of time if you debug a PR which is based on a different PR. The names are basically a concatenation of all elements inside the pipeline. Maybe it would actually help to have the classifier in the first place?

[joaquinvanschoren]

Well, I have some students trying it out right now so wanted to see if it can be easily fixed.

Turns out the KeyError is easily fixed.
in run.py, line 202, it checks for a component called 'Classifier', while it is called 'step__Classifier' in the flow. It's solved by replacing the line like this:

subflow = flow.components['steps__'+param]

But I guess you want to remove the 'steps__' prefix from the flow component names in the first place?

Anyway, this fixes the problem, but then another problem shows up. In the XML run description, the top-level workflow has a parameter 'Classifier', but in the uploaded flow there is only a parameter 'steps' with a JSON string: http://www.openml.org/f/4976

Does that make sense? I guess there is a problem with uploading the top-level flow?

[mfeurer]

I'm sorry to say, but pipelines should not be uploaded at all at the moment. There is no guarantee that one will be able to read the uploaded output in a week/month/year. Also, branches except the master branch should not be actively used. I'll try to have a look at the general problem this week (although I can't guarantee anything).