[MRG] Start organizing examples properly #30

jasmainak · 2018-09-15T03:16:41Z

Organizing the examples properly so that they can be built using sphinx gallery ...

jasmainak · 2018-09-15T03:17:58Z

examples/plot_power.py

+from params_power import main_path, data_path
+from params_power import subject_ids, sessions
+from params_power import power_analysis_name
+from params_power import fmin, fmax, power_method, is_epoched, freq_bands


@annapasca freq_bands was not imported in your example online. You can't detect such issues unless you do continuous integration ...

jasmainak · 2018-09-17T04:04:18Z

Hello @annapasca can you do the following:

$ git fetch upstream pull/30/head:fix_examples
$ git checkout fix_examples
$ cd doc/
$ make html

Are you able to make it work? For me, it doesn't work if I download the files that you shared on dropbox.
Simply doing:

%run examples/plot_01_preprocessing.py

in ipython should work but doesn't. I don't know why ...

Also, if you do:

$ make install

you should be able to push to http://neuropycon.github.io/ephypype and update the docs in one command. I also implemented automatic data fetcher. It's okay for now but the data is really big -- 3.6 GB!!! We should use a smaller dataset later. But one step at a time ...

annapasca · 2018-09-17T12:27:54Z

Hi @jasmainak !
I tried

$ git fetch upstream pull/30/head:fix_examples
$ git checkout fix_examples
$ cd doc/
$ make html

but I got the error

No module named 'ephypype.datasets'

jasmainak · 2018-09-17T14:28:02Z

If you do:

>>> from ephypype.datasets import fetch_omega_dataset

does it work? It should work ... do you do an editable install -- that is python setup.py develop?

codecov · 2018-09-19T05:35:50Z

Codecov Report

Merging #30 into master will decrease coverage by 1.21%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master      #30      +/-   ##
==========================================
- Coverage   29.19%   27.98%   -1.22%     
==========================================
  Files          19       19              
  Lines        1459     1458       -1     
==========================================
- Hits          426      408      -18     
- Misses       1033     1050      +17

Impacted Files	Coverage Δ
ephypype/power.py	`31.86% <ø> (-19.22%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ffe49e9...09f65ba. Read the comment docs.

jasmainak · 2018-09-19T05:49:08Z

@annapasca I managed to get the preprocessing and the power examples to "almost" work with sphinx gallery. Both of them get stuck -- and I don't know why. The issue of the missing datasets module should be fixed now. I had forgotten to commit that file. Can you try again?

$ git checkout fix_examples
$ git pull upstream pull/30/head
$ cd doc/
$ make html

Also, would you have a moment in the coming days to discuss on skype the API and simplifying the example? Maybe you can also help me understand why my examples get stuck ...

jasmainak · 2018-09-19T05:50:12Z

@EtienneCmb would you have some time to do the same as @annapasca ? To check that this PR works ...

annapasca · 2018-09-19T09:51:06Z

Hi @jasmainak!

I tried and make html works fine! The 2 workflows run properly.

I'm working in the mne env. Since I use anaconda3, your env is not fine for me.

I could install anaconda2 to see if also for me using your env I have the same problem.

What do you think?

annapasca · 2018-09-19T09:52:15Z

About the Skype, is it ok for you tomorrow 10 a.m. Mtl time?

jasmainak · 2018-09-19T13:20:52Z

Yep that works! Today?

On Wed 19 Sep 2018 at 05:52, Annalisa Pascarella ***@***.***> wrote: About the Skype, is it ok for you 10 a.m. Mtl time? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#30 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/APHiopWaVsC8TXvL7nU-rBRnVHGjEkpdks5uchPPgaJpZM4WqK9F> .

-- Sent from my iPhone

jasmainak · 2018-09-19T13:21:53Z

No I think I’m using MNE env now and it still doesn’t work ...

On Wed 19 Sep 2018 at 05:51, Annalisa Pascarella ***@***.***> wrote: Hi @jasmainak <https://github.com/jasmainak>! I tried and make html works fine! The 2 workflows run properly. I'm working in the mne env. Since I use anaconda3, your env is not fine for me. I could install anaconda2 to see if also for me using your env I have the same problem. How do you think? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#30 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/APHiolaPn6-BMATlK8PwQkDOQapJhNCxks5uchOXgaJpZM4WqK9F> .

-- Sent from my iPhone

annapasca · 2018-09-19T14:17:16Z

Yep that works! Today?

Tmw? :)

jasmainak · 2018-09-19T14:19:17Z

okay that's fine :) I saw your message from email first, so didn't see the edited message. So, we speak in 24 hours from now ...

annapasca · 2018-09-19T15:32:38Z

@jasmainak I changed 2 files to do new test on only 1 min of raw data (ICA seems to work).
What it the command to push on this PR? THKS A LOT!!!!!!!!!!!!!!!

I only changed in params.ica data_type=fif
and in plot_01_preprocessing.py datasource.inputs.template = '*%s/%s/meg/%s*rest*raw.fif'

jasmainak · 2018-09-19T15:37:43Z

You only changed the data right? It's on the same dropbox link so you don't need to push on this PR ... it should fetch the data automatically.

But if you want to know how to do it, you would do:

$ git remote add jasmainak https://github.com/jasmainak/ephypype
$ git remote push jasmainak fix_examples

EtienneCmb · 2018-09-19T15:58:58Z

plot_01_processing failed for me :

Unexpected failing examples:                                             │
/home/etienne/Toolbox/ephypype/examples/plot_01_preprocessing.py failed l
eaving traceback:                                                        │
Traceback (most recent call last):                                       │
  File "/home/etienne/Toolbox/ephypype/examples/plot_01_preprocessing.py"│
, line 100, in <module>                                                  │
    main_workflow.run(plugin='MultiProc', plugin_args={'n_procs': 3})    │
  File "/home/etienne/anaconda3/envs/pyqt5/lib/python3.6/site-packages/ni│
pype/pipeline/engine/workflows.py", line 595, in run                     │
    runner.run(execgraph, updatehash=updatehash, config=self.config)     │
  File "/home/etienne/anaconda3/envs/pyqt5/lib/python3.6/site-packages/ni│
pype/pipeline/plugins/base.py", line 192, in run                         │
    report_nodes_not_run(notrun)                                         │
  File "/home/etienne/anaconda3/envs/pyqt5/lib/python3.6/site-packages/ni│
pype/pipeline/plugins/tools.py", line 82, in report_nodes_not_run        │
    raise RuntimeError(('Workflow did not execute cleanly. '             │
RuntimeError: Workflow did not execute cleanly. Check log for details

jasmainak · 2018-09-19T15:59:44Z

Show me the full log?

jasmainak · 2018-09-19T16:00:22Z

You need to scroll up a bit to see the actual traceback ...

EtienneCmb · 2018-09-19T16:01:47Z

# Sphinx version: 1.7.5
# Python version: 3.6.1 (CPython)
# Docutils version: 0.14 
# Jinja2 version: 2.10
# Last messages:
#   dumping object inventory...
#   
#   done
#   
#   build succeeded, 12 warnings.
#   
#   
#   
#   The HTML pages are in _build/html.
#   
# Loaded extensions:
#   alabaster (0.7.10) from /home/etienne/anaconda3/envs/pyqt5/lib/python3.6/site-packages/alabaster/__init__.py
#   sphinx.ext.autodoc (1.7.5) from /home/etienne/anaconda3/envs/pyqt5/lib/python3.6/site-packages/sphinx/ext/autodoc/__init__.py
#   sphinx.ext.mathjax (1.7.5) from /home/etienne/anaconda3/envs/pyqt5/lib/python3.6/site-packages/sphinx/ext/mathjax.py
#   sphinx.ext.viewcode (1.7.5) from /home/etienne/anaconda3/envs/pyqt5/lib/python3.6/site-packages/sphinx/ext/viewcode.py
#   sphinx.ext.autosummary (1.7.5) from /home/etienne/anaconda3/envs/pyqt5/lib/python3.6/site-packages/sphinx/ext/autosummary/__init__.py
#   numpydoc (0.8.0) from /home/etienne/anaconda3/envs/pyqt5/lib/python3.6/site-packages/numpydoc/__init__.py
#   sphinx.ext.doctest (1.7.5) from /home/etienne/anaconda3/envs/pyqt5/lib/python3.6/site-packages/sphinx/ext/doctest.py
#   sphinx_gallery.gen_gallery (0.2.0) from /home/etienne/anaconda3/envs/pyqt5/lib/python3.6/site-packages/sphinx_gallery/gen_gallery.py
Traceback (most recent call last):
  File "/home/etienne/anaconda3/envs/pyqt5/lib/python3.6/site-packages/sphinx/cmdline.py", line 304, in main
    app.build(args.force_all, filenames)
  File "/home/etienne/anaconda3/envs/pyqt5/lib/python3.6/site-packages/sphinx/application.py", line 356, in build
    self.emit('build-finished', None)
  File "/home/etienne/anaconda3/envs/pyqt5/lib/python3.6/site-packages/sphinx/application.py", line 444, in emit
    return self.events.emit(event, self, *args)
  File "/home/etienne/anaconda3/envs/pyqt5/lib/python3.6/site-packages/sphinx/events.py", line 79, in emit
    results.append(callback(*args))
  File "/home/etienne/anaconda3/envs/pyqt5/lib/python3.6/site-packages/sphinx_gallery/gen_gallery.py", line 313, in sumarize_failing_examples
    "\n" + "-" * 79)
ValueError: Here is a summary of the problems encountered when running the examples

�[31;01mUnexpected failing examples:�[39;49;00m
/home/etienne/Toolbox/ephypype/examples/plot_01_preprocessing.py failed leaving traceback:
Traceback (most recent call last):
  File "/home/etienne/Toolbox/ephypype/examples/plot_01_preprocessing.py", line 100, in <module>
    main_workflow.run(plugin='MultiProc', plugin_args={'n_procs': 3})
  File "/home/etienne/anaconda3/envs/pyqt5/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 595, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/home/etienne/anaconda3/envs/pyqt5/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 192, in run
    report_nodes_not_run(notrun)
  File "/home/etienne/anaconda3/envs/pyqt5/lib/python3.6/site-packages/nipype/pipeline/plugins/tools.py", line 82, in report_nodes_not_run
    raise RuntimeError(('Workflow did not execute cleanly. '
RuntimeError: Workflow did not execute cleanly. Check log for details


-------------------------------------------------------------------------------

EtienneCmb · 2018-09-19T16:09:49Z

It should be specified at the top of your examples that some data are needed. I run the examples and it downloaded 675,9 Mo of data files...

jasmainak · 2018-09-19T16:23:20Z

It should be specified at the top of your examples that some data are needed. I run the examples and it downloaded 675,9 Mo of data files...

okay sure. I'm refactoring the example a bit so you will be able to soon see the fetch data command.

jasmainak · 2018-09-19T16:39:33Z

alright @EtienneCmb preprocessing example made more pedagogic. It could be made better, but it's a start ...

jasmainak · 2018-09-21T20:40:08Z

README.rst

@@ -116,7 +74,6 @@ Install dependencies with conda
 .. code-block:: bash 

    conda install pandas
-    conda install ipywidgets


@annapasca @dmalt I removed this. I think we discussed on hangouts that it is not used anywhere, right?

yes, for me it's ok!
If someone wants to inspect the ICA by notebook, he'll do it by himself.
What do u think @dmalt ?

oh sorry I didn't know that. Do you have the link for the notebook? So I can see what it is doing ...

https://neuropycon.github.io/neuropycon_doc/examples/preproc_example.html#preproc-example

at the bottom :)

jasmainak · 2018-09-21T20:41:01Z

README.rst

-
-see |MNE install| for more information.
-
-


@annapasca I also removed installation instruction for other packages. I think it's not so relevant here. We should simplify things to not bring people with MNE installation problems to neuropycon

ok! you are right! :)

jasmainak · 2018-09-21T21:10:56Z

doc/conf.py

+import os.path as op
+
+path = op.join(op.dirname(__file__), '../examples/')
+sys.path.insert(0, path)


@annapasca we should remove L201 and L202 after you reformat the power example ...

jasmainak · 2018-09-22T04:25:17Z

@annapasca I made CircleCI work now. I think this is good to go :)

Can you merge the pull request and work on top of it to improve the other examples? Waiting for your next pull request!

See the output here.

Couple of outstanding issues to be fixed later:

why is the output of nipype not being captured in the build ... (I don't know)
the examples should link or show the report that is produced
in the workflow graph, there is a node called preproc (mne). It looks like a black box to me. Are there more nodes inside this or is it a single node? Looking at it I can't tell if it's a single node

@dmalt I cleaned up the API page. However, as you mentioned today that there are other nodes which should be in public API, can you a make a pull request to add those?

I am going to focus on my other projects for the next couple of weeks. But I'll still look at the updates and pull requests. I hope we now have a good start and can keep the momentum going.

Cheers!

annapasca · 2018-09-22T07:50:57Z

Hi @jasmainak thanks a lot for this big review!
You left all in a way it's very clear and easy to continue :)
I have also noted you added function to create infosource and datasource. It was in my todo list, so thks also for this ;)

I'm going to merge all and next week I'll work on both example and unit test.

Since I'd like to learn how to be not too messy :D the right way after merge the PR is:

git checkout -b create_examples
git fetch upstream
git rebase upstream/master

I guess I will not be promoted :D :D :D

jasmainak · 2018-09-22T15:13:02Z

You should do:

$ git fetch upstream master:master
$ git checkout master
$ git branch my_feature_branch
$ git checkout my_feature_branch

jasmainak · 2018-09-22T15:13:15Z

Good luck :)

annapasca · 2018-09-23T06:49:05Z

You should do:

$ git fetch upstream master:master
$ git checkout master
$ git branch my_feature_branch
$ git checkout my_feature_branch

Grazieee!!! :) I have to study more ;)

jasmainak commented Sep 15, 2018

View reviewed changes

jasmainak force-pushed the fix_examples branch 2 times, most recently from 0b5a9b1 to 001b440 Compare September 15, 2018 03:36

jasmainak mentioned this pull request Sep 15, 2018

iterate on API + plans #32

Closed

jasmainak changed the title ~~[WIP] Start organizing examples properly~~ [MRG] Start organizing examples properly Sep 17, 2018

jasmainak force-pushed the fix_examples branch from f832d7e to 97171e8 Compare September 17, 2018 04:13

jasmainak mentioned this pull request Sep 17, 2018

how to run power example? #31

Closed

jasmainak force-pushed the fix_examples branch from 48ccddb to 52af9d0 Compare September 19, 2018 05:45

jasmainak force-pushed the fix_examples branch from b3cf79b to f9f4da4 Compare September 19, 2018 16:38

EtienneCmb and others added 5 commits September 21, 2018 16:16

Fix gitignore

bf78504

ENH: Fix path for new data

0883896

ENH: Simplify examples

8bcf7f6

MAINT: add circleci

1e3d7dc

Add an API page

2687152

jasmainak force-pushed the fix_examples branch from 24b6fe0 to 2687152 Compare September 21, 2018 20:16

Add CircleCI badge

3af52df

jasmainak force-pushed the fix_examples branch from c5197ad to 3af52df Compare September 21, 2018 20:24

DOC simplify installation instructions

ff1284f

jasmainak commented Sep 21, 2018

View reviewed changes

jasmainak force-pushed the fix_examples branch 4 times, most recently from 2641ff6 to 93ef662 Compare September 21, 2018 21:02

jasmainak commented Sep 21, 2018

View reviewed changes

FIX circleci build

3d0120f

jasmainak force-pushed the fix_examples branch 3 times, most recently from 61a28a6 to 4561c71 Compare September 22, 2018 03:40

FIX graphviz for viz of workflow

09f65ba

jasmainak force-pushed the fix_examples branch from 4561c71 to 09f65ba Compare September 22, 2018 03:46

annapasca merged commit 2b46f98 into neuropycon:master Sep 22, 2018

annapasca mentioned this pull request Sep 27, 2018

[WIP] Adding pipelines examples #41

Merged

jasmainak mentioned this pull request Sep 27, 2018

add circleci #33

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MRG] Start organizing examples properly #30

[MRG] Start organizing examples properly #30

jasmainak commented Sep 15, 2018

jasmainak Sep 15, 2018 •

edited

Loading

jasmainak commented Sep 17, 2018

annapasca commented Sep 17, 2018

jasmainak commented Sep 17, 2018

codecov bot commented Sep 19, 2018 •

edited

Loading

jasmainak commented Sep 19, 2018

jasmainak commented Sep 19, 2018

annapasca commented Sep 19, 2018 •

edited

Loading

annapasca commented Sep 19, 2018 •

edited

Loading

jasmainak commented Sep 19, 2018 via email

jasmainak commented Sep 19, 2018 via email

annapasca commented Sep 19, 2018

jasmainak commented Sep 19, 2018

annapasca commented Sep 19, 2018 •

edited

Loading

jasmainak commented Sep 19, 2018

EtienneCmb commented Sep 19, 2018

jasmainak commented Sep 19, 2018

jasmainak commented Sep 19, 2018

EtienneCmb commented Sep 19, 2018

EtienneCmb commented Sep 19, 2018

jasmainak commented Sep 19, 2018

jasmainak commented Sep 19, 2018

jasmainak Sep 21, 2018

annapasca Sep 22, 2018

jasmainak Sep 22, 2018

annapasca Sep 23, 2018

jasmainak Sep 21, 2018

annapasca Sep 22, 2018

jasmainak Sep 21, 2018

jasmainak commented Sep 22, 2018

annapasca commented Sep 22, 2018

jasmainak commented Sep 22, 2018

jasmainak commented Sep 22, 2018

annapasca commented Sep 23, 2018

[MRG] Start organizing examples properly #30

[MRG] Start organizing examples properly #30

Conversation

jasmainak commented Sep 15, 2018

jasmainak Sep 15, 2018 • edited Loading

Choose a reason for hiding this comment

jasmainak commented Sep 17, 2018

annapasca commented Sep 17, 2018

jasmainak commented Sep 17, 2018

codecov bot commented Sep 19, 2018 • edited Loading

Codecov Report

jasmainak commented Sep 19, 2018

jasmainak commented Sep 19, 2018

annapasca commented Sep 19, 2018 • edited Loading

annapasca commented Sep 19, 2018 • edited Loading

jasmainak commented Sep 19, 2018 via email

jasmainak commented Sep 19, 2018 via email

annapasca commented Sep 19, 2018

jasmainak commented Sep 19, 2018

annapasca commented Sep 19, 2018 • edited Loading

jasmainak commented Sep 19, 2018

EtienneCmb commented Sep 19, 2018

jasmainak commented Sep 19, 2018

jasmainak commented Sep 19, 2018

EtienneCmb commented Sep 19, 2018

EtienneCmb commented Sep 19, 2018

jasmainak commented Sep 19, 2018

jasmainak commented Sep 19, 2018

jasmainak Sep 21, 2018

Choose a reason for hiding this comment

annapasca Sep 22, 2018

Choose a reason for hiding this comment

jasmainak Sep 22, 2018

Choose a reason for hiding this comment

annapasca Sep 23, 2018

Choose a reason for hiding this comment

jasmainak Sep 21, 2018

Choose a reason for hiding this comment

annapasca Sep 22, 2018

Choose a reason for hiding this comment

jasmainak Sep 21, 2018

Choose a reason for hiding this comment

jasmainak commented Sep 22, 2018

annapasca commented Sep 22, 2018

jasmainak commented Sep 22, 2018

jasmainak commented Sep 22, 2018

annapasca commented Sep 23, 2018

jasmainak Sep 15, 2018 •

edited

Loading

codecov bot commented Sep 19, 2018 •

edited

Loading

annapasca commented Sep 19, 2018 •

edited

Loading

annapasca commented Sep 19, 2018 •

edited

Loading

annapasca commented Sep 19, 2018 •

edited

Loading