[FIX] PY3 compatibility revisited

[oesteban]

No description provided.

[satra]

accidental inclusion of this file

[coveralls]

Coverage decreased (-0.2%) to 72.041% when pulling f38fb07 on oesteban:fix/Py3UseBuiltinOpen into a26ef19 on nipy:master.

[oesteban]

Ok, we are growing too big here.

So, this is the situation: some parts of nipype are not python 3 compatible. The were passing unnoticed because the tests did not exercise these parts and CircleCI smoke tests were done in python 2.7.

In this PR, I changed the tests in CircleCI to run in python 3.5, with the following chain of issues:

• First problem were open() calls. I had to add a from builtins import open in all files using it.
• Now, these files would read/write unicode -> add from __future__ import unicode_literals.
• Many interfaces stopped working. Particularly those using Directory and DictStrStr. I fixed those, making Directory derive from traits.Unicode and DictStrStr with six.string_types. I also disabled the fast C type checker for them.
• I removed the external code of six. I don't see the point to maintain our version in parallel.

Finally, after too many modifications, the big issue is doctests. Python 2 will want to have a u-strings in tests, while Python 3 does not. There are two solutions:

1. Migrate to py.test (and enable ALLOW_UNICODE). However the test generators will not work, and we have plenty of those.
2. Tricks to fix the docstrings before nose:
   • https://dirkjan.ochtman.nl/writing/2014/07/06/single-source-python-23-doctests.html - Where should I place this code?
   • Use metaclasses and format the docstring before testing (the solution I'm testing right now, similar to the solution they deployed in IPython).
3. (EDITED) Use doctest-ignore-unicode 👍

Any opinions, @satra?

[coveralls]

Coverage decreased (-0.002%) to 72.269% when pulling 0921506 on oesteban:fix/Py3UseBuiltinOpen into 0f2cb9f on nipy:master.

[coveralls]

Coverage decreased (-0.002%) to 72.263% when pulling 38cbf46 on oesteban:fix/Py3UseBuiltinOpen into 4eb197f on nipy:master.

[coveralls]

Coverage decreased (-0.002%) to 72.263% when pulling 38cbf46 on oesteban:fix/Py3UseBuiltinOpen into 4eb197f on nipy:master.

[coveralls]

Coverage decreased (-0.002%) to 72.263% when pulling 38cbf46 on oesteban:fix/Py3UseBuiltinOpen into 4eb197f on nipy:master.

[coveralls]

Coverage decreased (-0.002%) to 72.263% when pulling 38cbf46 on oesteban:fix/Py3UseBuiltinOpen into 4eb197f on nipy:master.

[coveralls]

Coverage decreased (-0.002%) to 72.263% when pulling 38cbf46 on oesteban:fix/Py3UseBuiltinOpen into 4eb197f on nipy:master.

[coveralls]

Coverage decreased (-0.002%) to 72.263% when pulling 38cbf46 on oesteban:fix/Py3UseBuiltinOpen into 4eb197f on nipy:master.

[coveralls]

Coverage decreased (-0.002%) to 72.263% when pulling 38cbf46 on oesteban:fix/Py3UseBuiltinOpen into 4eb197f on nipy:master.

[coveralls]

Coverage decreased (-0.002%) to 72.263% when pulling 38cbf46 on oesteban:fix/Py3UseBuiltinOpen into 4eb197f on nipy:master.

[coveralls]

Coverage decreased (-0.002%) to 72.263% when pulling 644aa54 on oesteban:fix/Py3UseBuiltinOpen into 4eb197f on nipy:master.

[coveralls]

Coverage decreased (-0.002%) to 72.263% when pulling 644aa54 on oesteban:fix/Py3UseBuiltinOpen into 4eb197f on nipy:master.

[coveralls]

Coverage decreased (-0.002%) to 72.263% when pulling 644aa54 on oesteban:fix/Py3UseBuiltinOpen into 4eb197f on nipy:master.

[coveralls]

Coverage decreased (-0.002%) to 72.263% when pulling 644aa54 on oesteban:fix/Py3UseBuiltinOpen into 4eb197f on nipy:master.

[coveralls]

Coverage decreased (-0.002%) to 72.263% when pulling 644aa54 on oesteban:fix/Py3UseBuiltinOpen into 4eb197f on nipy:master.

[coveralls]

Coverage decreased (-0.002%) to 72.263% when pulling 644aa54 on oesteban:fix/Py3UseBuiltinOpen into 4eb197f on nipy:master.

[coveralls]

Coverage decreased (-0.002%) to 72.263% when pulling 644aa54 on oesteban:fix/Py3UseBuiltinOpen into 4eb197f on nipy:master.

[coveralls]

Coverage decreased (-0.002%) to 72.263% when pulling 644aa54 on oesteban:fix/Py3UseBuiltinOpen into 4eb197f on nipy:master.

[oesteban]

this is finally ready!. It is very worrying that the MultiProc plugin is about twice as slow as the Linear plugin... I have to disable the resource profiler to discard it as bottleneck (in another PR).

[satra]

the resource profiler should be disabled in current master

[satra]

shall we merge this? it seems like a lot of things are dependent on this now.

[oesteban]

@satra - I think so. Regarding the resource profiler: I will investigate the issue. Do you think that could be the explanation for MultiProc taking so much longer?

[chrisgorgo]

Where is the NIPYPE_NUMBER_OF_CPUS variable used?

[oesteban]

https://github.com/oesteban/nipype/blob/fix/Py3UseBuiltinOpen/tools/run_examples.py#L23

[chrisgorgo]

I see!

On Wed, Sep 7, 2016 at 11:41 AM, Oscar Esteban notifications@github.com
wrote:

In circle.yml
#1572 (comment):

     timeout: 1600

• • docker run -v /etc/localtime:/etc/localtime:ro -v ~/examples:/root/examples:ro -v $(pwd)/scratch:/scratch -w /scratch nipype/nipype_test:py27 /usr/bin/run_examples.sh fmri_fsl_reuse Linear /root/examples/ level1_workflow
• • docker run -v /etc/localtime:/etc/localtime:ro -v ~/examples:/root/examples:ro -v $(pwd)/scratch:/scratch -w /scratch nipype/nipype_test:py27 /usr/bin/run_examples.sh fmri_spm_nested Linear /root/examples/ level1
• • docker run -v /etc/localtime:/etc/localtime:ro -v ~/examples:/root/examples:ro -v $(pwd)/scratch:/scratch -w /scratch nipype/nipype_test:py27 /usr/bin/run_examples.sh fmri_spm_nested Linear /root/examples/ l2pipeline
• • docker run -e FSL_COURSE_DATA="/root/examples/nipype-fsl_course_data" -v /etc/localtime:/etc/localtime:ro -v ~/scratch:/scratch -w /scratch nipype/nipype_test:py27 /usr/bin/run_nosetests.sh :

•    timeout: 2600
  

• • docker run -v /etc/localtime:/etc/localtime:ro -v ~/examples:/root/examples:ro -v ~/scratch:/scratch -w /scratch nipype/nipype_test:py35 /usr/bin/run_examples.sh fmri_fsl_reuse Linear /root/examples/ level1_workflow
• • docker run -v /etc/localtime:/etc/localtime:ro -e NIPYPE_NUMBER_OF_CPUS=4 -v ~/examples:/root/examples:ro -v ~/scratch:/scratch -w /scratch nipype/nipype_test:py27 /usr/bin/run_examples.sh fmri_spm_nested MultiProc /root/examples/ level1

https://github.com/oesteban/nipype/blob/fix/Py3UseBuiltinOpen/tools/run_
examples.py#L23

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nipy/nipype/pull/1572/files/644aa549fd478afb50419774e3238eab0c08ab93#r77880468,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOkp4eDjPttM0KyEuGRHIgJO0KyZRFRks5qnwV0gaJpZM4Jfl9M
.

[oesteban]

It was the way to set it in run_examples.py with the least changes.