[MRG+2] use distinct filenames for pickled datasets under Python 2 and 3

[ogrisel]

This is a fix for #3596.

Note that I changed the covtype loader to be consistent with the others even though that might re-trigger a useless download once.

[ogrisel]

Hum, I will fix the issue under Python 2.

[ogrisel]

The what's new entry is conflicting. As it's going to change a lot today, I will do a final rebase once the tests are all green and reviewers are ok with the change.

[jnothman]

Is it possible to just use the lowest pickle version setting? Or is the issue that the constructors differ?

Should we anticipate different Py3 versions producing mutually incompatible pickles? Someone may have multiple versions of Py3 installed in different venvs but share the same default data dir.

[TomDLT]

LGTM

[ogrisel]

Is it possible to just use the lowest pickle version setting? Or is the issue that the constructors differ?

This is what @lesteve tried to do in joblib but there is still issues with the fact that the numpy pickler used in some cases (small arrays with compression) tries to represent the buffer content with a str instance under Python 2 (which is fine) but str instance under Python 3 imply decoding. They should be unpickled as bytes under Python 3 but only for str instances that represent the binary content of a numpy buffer. This is very complex to get right and I feel that is going to be a serious maintenance headache.

I prefer to make it explicit that we expect pickles not to be compatible across versions of Python.

[ogrisel]

Should we anticipate different Py3 versions producing mutually incompatible pickles? Someone may have multiple versions of Py3 installed in different venvs but share the same default data dir.

That should not be a problem as the str type behavior should not change again under Python 3.

[jnothman]

Okay.

On 7 October 2015 at 21:51, Olivier Grisel notifications@github.com wrote:

Should we anticipate different Py3 versions producing mutually
incompatible pickles? Someone may have multiple versions of Py3 installed
in different venvs but share the same default data dir.

That should not be a problem as the str type behavior should not change
again under Python 3.

—
Reply to this email directly or view it on GitHub
#5355 (comment)
.

[jnothman]

What's going on here? Does this fix make a practical difference that should be recorded as a bug fix?

[ogrisel]

It was to fix a numpy warning that does not like indexing by float values. The value is 128.0. I do not really understand this code but that should not impact the behavior besides the warning.

[jnothman]

Ah of course

[jnothman]

LGTM

[ogrisel]

Ok I will merge by rebase then.

[jnothman]

Ta

[amueller]

Maybe it would also be a good idea to include joblib and scikit-learn version in the files. That might be more insight the pickle, though? We had some issues with changes in scikit-learn already.