Modifications for building docs with python3 #1091

ghoshbishakh · 2016-06-28T10:08:31Z

Change apigen.py for python3 support
And minor PEP8 changes
fixes #912

coveralls · 2016-06-28T10:53:15Z

Coverage remained the same at 82.88% when pulling b15714b on ghoshbishakh:docsPy3 into 220d883 on nipy:master.

codecov-io · 2016-06-28T10:58:51Z

Current coverage is 80.84% (diff: 100%)

Merging #1091 into master will not change coverage

@@             master      #1091   diff @@
==========================================
  Files           200        200          
  Lines         23023      23023          
  Methods           0          0          
  Messages          0          0          
  Branches       2309       2309          
==========================================
  Hits          18614      18614          
  Misses         3933       3933          
  Partials        476        476

Powered by Codecov. Last update 63f6a5f...37fc41e

coveralls · 2016-06-28T11:38:06Z

Coverage remained the same at 82.88% when pulling b15714b on ghoshbishakh:docsPy3 into 220d883 on nipy:master.

arokem · 2016-07-01T14:31:25Z

Nice! But documentation still does not build on Python 3 with this fix. Still need to also adjust this file to be Python 3 compliant: https://github.com/nipy/dipy/blob/master/doc/sphinxext/github.py

This line, for starters: https://github.com/nipy/dipy/blob/master/doc/sphinxext/github.py#L39

(that's where I get an error raised)

ghoshbishakh · 2016-07-01T14:41:34Z

I don't get it on make api or make html Any suggestions how should I test it?

arokem · 2016-07-01T15:03:15Z

I am not sure. Could it be that your system is getting the github sphinx extension from somewhere else?

arokem · 2016-07-01T15:04:32Z

Either way, it's still not Python 3 compliant.

Try:

cd ~/source/dipy/doc/sphinxext
python -c"import github"

ghoshbishakh · 2016-07-01T15:05:46Z

@arokem yes I will fix that file. I have to check why I am not getting the error.

ghoshbishakh · 2016-07-01T15:35:16Z

@arokem sorry my bad. I was using python2 sphinx as default. I found that numpydoc extension also crashes. I think removing it from the sphinxext folder will be better as we can use the lastest numpydoc without shipping an older version. What do you recommend?

arokem · 2016-07-01T15:40:02Z

You can also get python 3 compliant versions of all these here: https://github.com/uwescience/shablona/tree/master/doc/sphinxext

ghoshbishakh · 2016-07-01T15:42:20Z

@arokem okey then I will include those.

coveralls · 2016-07-01T16:09:31Z

Coverage increased (+0.03%) to 82.908% when pulling 40790b7 on ghoshbishakh:docsPy3 into 220d883 on nipy:master.

Garyfallidis · 2016-07-12T08:49:57Z

Looks good. Thanks @ghoshbishakh
Is there anything left on this PR? If not I will be happy to merge it.

ghoshbishakh · 2016-07-12T09:05:14Z

@Garyfallidis Not much is left. But I have not been able to test it completely with python3 as it takes a long time. You may merge it as I can make patches later also.
By completely I mean with all examples.

Garyfallidis · 2016-07-12T09:28:54Z

Why does it take a long time?

ghoshbishakh · 2016-07-12T09:30:43Z

@Garyfallidis Because I have only 4GB RAM and the build process requires about 16GB. And using swap it takes about 20 hours to build the entire doc.

Garyfallidis · 2016-07-12T09:54:31Z

I think there are only a few examples that need more memory than usual. DKI and Life. You may want to disable them by commenting them at doc/examples/valid_examples.txt
Here is what I think may require more memory.
linear_fascicle_evaluation.py
reconst_dki.py
Let me know if that helped and if you can now compile the documentation faster.

ghoshbishakh · 2016-07-12T10:17:33Z

@Garyfallidis ok. I will build it and post a link here.

ghoshbishakh · 2016-07-13T10:12:14Z

I am getting this error:

segment_extending_clustering_framework.py
segment_clustering_features.py
Nb. clusters: 4
Cluster sizes: <map object at 0x7fa07c621588>
Nb. clusters: 4
Cluster sizes: <map object at 0x7fa07d5eb3c8>
Traceback (most recent call last):
  File "../tools/make_examples.py", line 122, in <module>
    exec(open(script).read(), namespace)
  File "<string>", line 172, in <module>
IndexError: too many indices for array
Makefile:122: recipe for target 'rstexamples-stamp' failed
make: *** [rstexamples-stamp] Error 1
make html  1442.06s user 18.58s system 134% cpu 18:07.87 total

Garyfallidis · 2016-07-20T14:00:08Z

Hi @MarcCote have you seen this? Any ideas?

ghoshbishakh · 2016-07-21T09:12:42Z

I ran the code for segment_clustering_features.py separately and I get this:

python segment_clustering_features.py 
Nb. clusters: 4
Cluster sizes: <map object at 0x7faf5bcc55f8>
Nb. clusters: 4
Cluster sizes: <map object at 0x7faf5df22358>
Traceback (most recent call last):
  File "segment_clustering_features.py", line 172, in <module>
    fvtk.add(ren, fvtk.point(centers[:, 0, :], colormap_full, point_radius=0.2))
IndexError: too many indices for array

MarcCote · 2016-07-22T15:29:17Z

@ghoshbishakh: @Garyfallidis just told me about your issue with segment_clustering_features.py. I just made a PR on your branch. That should resolve the issue. Basically, since in Python3 map(...) returns a generator, I had to cast them into list objects.

ghoshbishakh · 2016-07-23T13:15:07Z

What did just happen? I merged @MarcCote 's PR and pulled it locally and then made a push and this happened. 😟

ghoshbishakh · 2016-07-23T13:20:04Z

@MarcCote seems I have made a successful reset! Can you somehow make the PR again?

coveralls · 2016-07-23T13:26:05Z

Coverage remained the same at 82.908% when pulling 40790b7 on ghoshbishakh:docsPy3 into c09a194 on nipy:master.

coveralls · 2016-07-23T14:18:27Z

Coverage remained the same at 82.908% when pulling 40790b7 on ghoshbishakh:docsPy3 into c09a194 on nipy:master.

arokem · 2016-07-23T19:43:03Z

You might want to both be rebased on top of current master before trying
again :-)

On Sat, Jul 23, 2016 at 7:18 AM, Coveralls notifications@github.com wrote:

[image: Coverage Status] https://coveralls.io/builds/7131096

Coverage decreased (-0.4%) to 82.474% when pulling 40790b7
40790b7
on ghoshbishakh:docsPy3 into c09a194
c09a194
on nipy:master.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1091 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAHPNkVJg8_Or_nxifCSqeJD6Mh52b0cks5qYiK0gaJpZM4I_8U0
.

ghoshbishakh · 2016-07-24T05:19:23Z

As @arokem suggested, I rebased on master and made a push. @MarcCote can you send the PR now? Lets see if it works out properly this time.

coveralls · 2016-07-24T05:32:21Z

Coverage remained the same at 82.908% when pulling 2ebe0f6 on ghoshbishakh:docsPy3 into c09a194 on nipy:master.

coveralls · 2016-07-24T13:28:39Z

Coverage remained the same at 82.908% when pulling 0775018 on ghoshbishakh:docsPy3 into c09a194 on nipy:master.

ghoshbishakh · 2016-07-24T14:59:16Z

That PR solved that issue @MarcCote 😄
But now I am getting another error:

Computing tensors...
Computing worst-case/best-case SNR using the corpus callosum...
Noise standard deviation sigma=  9.83112897825
SNR for the b=0 image is : 42.0959119804
SNR for direction 58   [ 0.98875  0.1177  -0.09229] is : 5.47338194103
SNR for direction 57   [-0.05039    0.99871    0.0054406] is : 23.9479488745
SNR for direction 126   [-0.11825  -0.039925  0.99218 ] is : 24.0116089426
streamline_formats.py
/home/bishakh/projects/dipy/dipy/data/files/tracks300.trk
Traceback (most recent call last):
  File "../tools/make_examples.py", line 122, in <module>
    exec(open(script).read(), namespace)
  File "<string>", line 54, in <module>
  File "/home/bishakh/projects/dipy/dipy/io/dpy.py", line 70, in __init__
    ['0.0.1'], 'Dpy Version Number')
  File "/usr/lib/python3.5/site-packages/tables/_past.py", line 35, in oldfunc
    return obj(*args, **kwargs)
  File "/usr/lib/python3.5/site-packages/tables/file.py", line 1152, in create_array
    obj=obj, title=title, byteorder=byteorder)
  File "/usr/lib/python3.5/site-packages/tables/array.py", line 188, in __init__
    byteorder, _log)
  File "/usr/lib/python3.5/site-packages/tables/leaf.py", line 262, in __init__
    super(Leaf, self).__init__(parentnode, name, _log)
  File "/usr/lib/python3.5/site-packages/tables/node.py", line 267, in __init__
    self._v_objectid = self._g_create()
  File "/usr/lib/python3.5/site-packages/tables/array.py", line 205, in _g_create
    raise TypeError("Array objects cannot currently deal with void, "
TypeError: Array objects cannot currently deal with void, unicode or object arrays
Closing remaining open files:fornix.dpy...done
Makefile:122: recipe for target 'rstexamples-stamp' failed
make: *** [rstexamples-stamp] Error 1
make html  1588.85s user 1668.44s system 216% cpu 25:02.59 total

ghoshbishakh · 2016-07-24T15:03:07Z

Here is the output if I run streamline_formats.py separately:

bishakh@bisu-arch ~/projects/dipy/doc/examples (git)-[docsPy3] % python streamline_formats.py 
/home/bishakh/projects/dipy/dipy/data/files/tracks300.trk
Traceback (most recent call last):
  File "streamline_formats.py", line 54, in <module>
    dpw = Dpy('fornix.dpy', 'w')
  File "/home/bishakh/projects/dipy/dipy/io/dpy.py", line 70, in __init__
    ['0.0.1'], 'Dpy Version Number')
  File "/usr/lib/python3.5/site-packages/tables/_past.py", line 35, in oldfunc
    return obj(*args, **kwargs)
  File "/usr/lib/python3.5/site-packages/tables/file.py", line 1152, in create_array
    obj=obj, title=title, byteorder=byteorder)
  File "/usr/lib/python3.5/site-packages/tables/array.py", line 188, in __init__
    byteorder, _log)
  File "/usr/lib/python3.5/site-packages/tables/leaf.py", line 262, in __init__
    super(Leaf, self).__init__(parentnode, name, _log)
  File "/usr/lib/python3.5/site-packages/tables/node.py", line 267, in __init__
    self._v_objectid = self._g_create()
  File "/usr/lib/python3.5/site-packages/tables/array.py", line 205, in _g_create
    raise TypeError("Array objects cannot currently deal with void, "
TypeError: Array objects cannot currently deal with void, unicode or object arrays
Closing remaining open files:fornix.dpy...done

ghoshbishakh · 2016-07-24T19:48:58Z

looks like changing

self.version = self.f.createArray(self.f.root, 'version',
                                              ['0.0.1'], 'Dpy Version Number')

to

self.version = self.f.createArray(self.f.root, 'version',
                                              [b"0.0.1"], 'Dpy Version Number')

fixes it.

Got some clue from here:
PyTables/PyTables@b7c1430

ghoshbishakh · 2016-07-27T13:31:34Z

@arokem can you suggest me if I should make that change in the string in this PR?

arokem · 2016-07-27T14:09:08Z

Where is this change implemented? In dipy/io/dpy.py? Does it work with Python 2 when you implement this change?

ghoshbishakh · 2016-07-28T15:32:46Z

@arokem yes the change is in dipy/io/dpy.py

And yes it works in both python2 and 3 and the output is:

bishakh@bisu-arch ~/projects/dipy/doc/examples (git)-[docsPy3] % python2 streamline_formats.py
/home/bishakh/projects/dipy/dipy/data/files/tracks300.trk
300
5
bishakh@bisu-arch ~/projects/dipy/doc/examples (git)-[docsPy3] % python streamline_formats.py 
/home/bishakh/projects/dipy/dipy/data/files/tracks300.trk
300
5

arokem · 2016-07-28T15:34:55Z

Then yes - I say we should include it in this PR. @Garyfallidis : any thoughts here?

arokem · 2016-07-28T15:35:53Z

Sorry - let me walk back that last comment. Thinking about this for one more second: I think this should be a separate PR. In particular, I would like to see a test that fails on current master, and is fixed when this change is introduced.

arokem · 2016-07-28T15:36:28Z

That should make for a really small PR, which we can merge quickly. Then, you can rebase this current PR on top of that change and move ahead with it. How does that sound?

ghoshbishakh · 2016-07-28T15:39:21Z

@arokem Ok I am making a separate PR quickly

arokem · 2016-07-29T14:09:23Z

OK - now that #1100 is merged, you can rebase this one and keep going here.

And minor PEP8 changes fixes dipy#912

@arokem

Taken from @arokem 's commit: uwescience/shablona@ee63354

coveralls · 2016-07-29T17:14:43Z

Coverage remained the same at 82.908% when pulling 37fc41e on ghoshbishakh:docsPy3 into 63f6a5f on nipy:master.

ghoshbishakh · 2016-07-31T05:40:09Z

Now there is a problem with restore_dti.py

restore_dti.py
Traceback (most recent call last):
  File "../tools/make_examples.py", line 122, in <module>
    exec(open(script).read(), namespace)
  File "<string>", line 39, in <module>
NameError: name 'reload' is not defined
make: *** [Makefile:122: rstexamples-stamp] Error 1
make html  6893.82s user 29.25s system 244% cpu 47:13.14 total

ghoshbishakh · 2016-07-31T08:39:43Z

except for issue #1104 everything works fine now! I have made a PR #1105 to solve that.

ghoshbishakh · 2016-07-31T17:38:05Z

@arokem now this works with all examples enabled except :

linear_fascicle_evaluation.py
reconst_dki.py

I have not checked for these two

arokem · 2016-07-31T18:11:51Z

Why are you not running these? The DKI example does rely on downloading a large amount of data (we need data with multiple b-values...). I just ran it on Python 3 on my laptop and it runs fine.

The LiFE example (linear_fascicle_evalutation.py) doesn't require large amounts of data, or large amounts of RAM, so you should try running it. I can't get it to run on my machine, because I don't have VTK installed on my Python 3 installation, but if you have VTK installed, you should try running that.

ghoshbishakh · 2016-07-31T18:33:16Z

I ran LiFE example and it does take a lot of ram and my pc is stuck now. I
will let it run for tonight and check the result tomorrow morning.

Just out of curiosity.. I noticed some examples use all the 4 cores to
process and run considerably fast while some others make use of only one
core and take a lot of time.
Why is this and can this be improved?

On Jul 31, 2016 11:41 PM, "Ariel Rokem" notifications@github.com wrote:

Why are you not running these? The DKI example does rely on downloading a
large amount of data (we need data with multiple b-values...). I just ran
it on Python 3 on my laptop and it runs fine.

The LiFE example (linear_fascicle_evalutation.py) doesn't require large
amounts of data, or large amounts of RAM, so you should try running it. I
can't get it to run on my machine, because I don't have VTK installed on my
Python 3 installation, but if you have VTK installed, you should try
running that.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1091 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AE6vs6vhJ3dcYDiEp0GIcD6h159qn9JCks5qbOVpgaJpZM4I_8U0
.

arokem · 2016-07-31T18:48:51Z

Did you run the streamline_tools example beforehand?

If you have the file 'lr-superiorfrontal.trk' in place already (that is,
after tracking) the LiFE example takes (on my machine) about 3 minutes to
run, and takes about 38 MB RAM.

As for full utilization of cpu: some of our code already uses
multiprocessing, and some still needs to be rewritten to use
multiprocessing :-)

On Sun, Jul 31, 2016 at 11:33 AM, Bishakh Ghosh notifications@github.com
wrote:

I ran LiFE example and it does take a lot of ram and my pc is stuck now. I
will let it run for tonight and check the result tomorrow morning.

Just out of curiosity.. I noticed some examples use all the 4 cores to
process and run considerably fast while some others make use of only one
core and take a lot of time.
Why is this and can this be improved?

On Jul 31, 2016 11:41 PM, "Ariel Rokem" notifications@github.com wrote:

Why are you not running these? The DKI example does rely on downloading a
large amount of data (we need data with multiple b-values...). I just ran
it on Python 3 on my laptop and it runs fine.

The LiFE example (linear_fascicle_evalutation.py) doesn't require large
amounts of data, or large amounts of RAM, so you should try running it. I
can't get it to run on my machine, because I don't have VTK installed on
my
Python 3 installation, but if you have VTK installed, you should try
running that.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1091 (comment), or mute
the thread
<
https://github.com/notifications/unsubscribe-auth/AE6vs6vhJ3dcYDiEp0GIcD6h159qn9JCks5qbOVpgaJpZM4I_8U0

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1091 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAHPNo7mIEWi3jr1iKPVejbaXXvP_drWks5qbOptgaJpZM4I_8U0
.

ghoshbishakh · 2016-08-01T02:05:58Z

@arokem I just ran the example separately this time and it worked.

bishakh@bisu-arch ~/projects/dipy/doc/examples (git)-[docsPy3] % python linear_fascicle_evaluation.py
Dataset is already in place. If you want to fetch it again please first remove the folder /home/bishakh/.dipy/stanford_hardi 
Dataset is already in place. If you want to fetch it again please first remove the folder /home/bishakh/.dipy/stanford_hardi 
Dataset is already in place. If you want to fetch it again please first remove the folder /home/bishakh/.dipy/stanford_hardi 
Dataset is already in place. If you want to fetch it again please first remove the folder /home/bishakh/.dipy/stanford_hardi 
Dataset is already in place. If you want to fetch it again please first remove the folder /home/bishakh/.dipy/stanford_hardi 
Camera Position (40.00,52.50,330.80)
Camera Focal Point (40.00,52.50,37.50)
Camera View Up (0.00,1.00,0.00)
-------------------------------------
Camera New Position (-1.00,0.00,0.00)
Camera New Focal Point (0.00,0.00,0.00)
Camera New View Up (0.00,0.00,1.00)
python linear_fascicle_evaluation.py  285.87s user 16.08s system 19% cpu 25:44.50 total

arokem · 2016-08-01T02:27:10Z

Excellent! I think this is probably ready to go then, right?

Anyone else want to take a look?

ghoshbishakh · 2016-08-01T06:17:22Z

@arokem yes it is working perfectly. Let others take a look for a couple of days.

ghoshbishakh changed the title ~~Modifications in apigen.py for building docs with python3~~ Modifications for building docs with python3 Jun 28, 2016

MarcCote mentioned this pull request Jul 22, 2016

FIX: convert map to list (Python3 compatibility) ghoshbishakh/dipy#3

Merged

ghoshbishakh force-pushed the docsPy3 branch from 0174213 to 40790b7 Compare July 23, 2016 13:18

ghoshbishakh and others added 4 commits July 29, 2016 20:05

Change apigen.py for python3 support

d9425d6

And minor PEP8 changes fixes dipy#912

Modify make_examples.py for python3 support

29e9baa

Update sphinx extensions

1f8d39d

Taken from @arokem 's commit: uwescience/shablona@ee63354

FIX: convert map to list (Python3 compatibility)

37fc41e

ghoshbishakh force-pushed the docsPy3 branch from 0775018 to 37fc41e Compare July 29, 2016 14:35

arokem merged commit e309c15 into dipy:master Aug 2, 2016

Modifications for building docs with python3 #1091

Modifications for building docs with python3 #1091

Conversation

ghoshbishakh commented Jun 28, 2016

coveralls commented Jun 28, 2016 • edited Loading

codecov-io commented Jun 28, 2016 • edited Loading

Current coverage is 80.84% (diff: 100%)

coveralls commented Jun 28, 2016 • edited Loading

arokem commented Jul 1, 2016

ghoshbishakh commented Jul 1, 2016 via email

arokem commented Jul 1, 2016

arokem commented Jul 1, 2016

ghoshbishakh commented Jul 1, 2016

ghoshbishakh commented Jul 1, 2016

arokem commented Jul 1, 2016

ghoshbishakh commented Jul 1, 2016

coveralls commented Jul 1, 2016 • edited Loading

Garyfallidis commented Jul 12, 2016

ghoshbishakh commented Jul 12, 2016 • edited Loading

Garyfallidis commented Jul 12, 2016

ghoshbishakh commented Jul 12, 2016

Garyfallidis commented Jul 12, 2016 • edited Loading

ghoshbishakh commented Jul 12, 2016

ghoshbishakh commented Jul 13, 2016

Garyfallidis commented Jul 20, 2016

ghoshbishakh commented Jul 21, 2016

MarcCote commented Jul 22, 2016

ghoshbishakh commented Jul 23, 2016

ghoshbishakh commented Jul 23, 2016

coveralls commented Jul 23, 2016 • edited Loading

coveralls commented Jul 23, 2016 • edited Loading

arokem commented Jul 23, 2016

ghoshbishakh commented Jul 24, 2016

coveralls commented Jul 24, 2016 • edited Loading

coveralls commented Jul 24, 2016 • edited Loading

ghoshbishakh commented Jul 24, 2016

ghoshbishakh commented Jul 24, 2016

ghoshbishakh commented Jul 24, 2016

ghoshbishakh commented Jul 27, 2016

arokem commented Jul 27, 2016

ghoshbishakh commented Jul 28, 2016

arokem commented Jul 28, 2016

arokem commented Jul 28, 2016

arokem commented Jul 28, 2016

ghoshbishakh commented Jul 28, 2016

arokem commented Jul 29, 2016

coveralls commented Jul 29, 2016 • edited Loading

ghoshbishakh commented Jul 31, 2016

ghoshbishakh commented Jul 31, 2016

ghoshbishakh commented Jul 31, 2016 • edited Loading

arokem commented Jul 31, 2016

ghoshbishakh commented Jul 31, 2016

arokem commented Jul 31, 2016

ghoshbishakh commented Aug 1, 2016

arokem commented Aug 1, 2016

ghoshbishakh commented Aug 1, 2016

coveralls commented Jun 28, 2016 •

edited

Loading

codecov-io commented Jun 28, 2016 •

edited

Loading

coveralls commented Jun 28, 2016 •

edited

Loading

coveralls commented Jul 1, 2016 •

edited

Loading

ghoshbishakh commented Jul 12, 2016 •

edited

Loading

Garyfallidis commented Jul 12, 2016 •

edited

Loading

coveralls commented Jul 23, 2016 •

edited

Loading

coveralls commented Jul 23, 2016 •

edited

Loading

coveralls commented Jul 24, 2016 •

edited

Loading

coveralls commented Jul 24, 2016 •

edited

Loading

coveralls commented Jul 29, 2016 •

edited

Loading

ghoshbishakh commented Jul 31, 2016 •

edited

Loading