ci: deploy docs from CI pipeline

[ErikBjare]

Builds upon work by @Div12345 in #172

Partially fixes #78

[ErikBjare]

I'll squash and merge this once I have it working and passing.

[ErikBjare]

Building the docs is taking a surprisingly long time...

[ErikBjare]

Looks like a couple examples fail: https://github.com/NeuroTechX/moabb/runs/2234979328

Extension error:
Here is a summary of the problems encountered when running the examples

Unexpected failing examples:
/home/runner/work/moabb/moabb/examples/plot_cross_session_multiple_datasets.py failed leaving traceback:
Traceback (most recent call last):
  File "/home/runner/work/moabb/moabb/examples/plot_cross_session_multiple_datasets.py", line 85, in <module>
    X, labels, meta = paradigm.get_data(dataset=d, subjects=[2])
  File "/home/runner/work/moabb/moabb/moabb/paradigms/base.py", line 262, in get_data
    metadata = pd.concat(metadata, ignore_index=True)
  File "/home/runner/.cache/pypoetry/virtualenvs/moabb-gXfHUFBj-py3.6/lib/python3.6/site-packages/pandas/core/reshape/concat.py", line 284, in concat
    sort=sort,
  File "/home/runner/.cache/pypoetry/virtualenvs/moabb-gXfHUFBj-py3.6/lib/python3.6/site-packages/pandas/core/reshape/concat.py", line 331, in __init__
    raise ValueError("No objects to concatenate")
ValueError: No objects to concatenate


/home/runner/work/moabb/moabb/examples/plot_cross_session_ssvep.py failed leaving traceback:
Traceback (most recent call last):
  File "/home/runner/work/moabb/moabb/examples/plot_cross_session_ssvep.py", line 81, in <module>
    X, labels, meta = paradigm.get_data(dataset=dataset, subjects=[3])
  File "/home/runner/work/moabb/moabb/moabb/paradigms/base.py", line 262, in get_data
    metadata = pd.concat(metadata, ignore_index=True)
  File "/home/runner/.cache/pypoetry/virtualenvs/moabb-gXfHUFBj-py3.6/lib/python3.6/site-packages/pandas/core/reshape/concat.py", line 284, in concat
    sort=sort,
  File "/home/runner/.cache/pypoetry/virtualenvs/moabb-gXfHUFBj-py3.6/lib/python3.6/site-packages/pandas/core/reshape/concat.py", line 331, in __init__
    raise ValueError("No objects to concatenate")
ValueError: No objects to concatenate


-------------------------------------------------------------------------------
dumping object inventory... done
build succeeded, 42 warnings.

The HTML pages are in build/html.

Sphinx-Gallery successfully executed 7 out of 9 files subselected by:

    gallery_conf["filename_pattern"] = '\\/plot'
    gallery_conf["ignore_pattern"]   = '__init__\\.py'

after excluding 0 files that had previously been run (based on MD5).

Makefile:20: recipe for target 'html' failed

[ErikBjare]

@sylvchev Do you know what is going wrong in the two failing examples?

[sylvchev]

I'll take a look at it right now
The doc is long to build because it need to download part of the dataset. Even when trying to keep it low, this is really long.

[ErikBjare]

@sylvchev Ah, should probably set up caching for the datasets then. Where are they stored after being downloaded?

[Div12345]

Just added a bit to the assertion error warning to figure out what is happening.. https://github.com/NeuroTechX/moabb/pull/174/checks?check_run_id=2732447233#step:7:520
AssertionError: provided order does not have all columns of dataframe, order = {'RG+LDA', 'CSP+LDA', 'RG+LR', 'CSP+LR'}, dfcols = {'CSP+LDA'}

[sylvchev]

I think I found a solution for the failing doc building. The results of examples and tutorials are sometime cached or recomputed, depending on the overwrite parameter of the evaluation. I added a different suffix (using suffix parameter) to differentiate between those cached results. We still need to run some tests to verify that there is no bug behind: a result with a specific suffix (suffix=whatever) that is not overwritten (overwrite=False) should be recomputed only for new pipelines/datasets/evaluation type (within-session/cross-session/cross-subject) with the same suffix.

To do:

• Move select_electrodes_resample.py, plot_explore_paradigm.py and plot_statistical_analysis.py from tutorials to examples (and rename explore paradigm, as there is not plot I think)
• Check warning in building docs (WARNING: duplicate object description, Bullet list ends without a blank line; unexpected unindent, Unknown target name, ...)
• it is very suspicious that some of the examples/tutorials are generated in 0.0 s (examples/mne_and_scikit_estimators.py, select_electrodes_resample.py and all tutorial_X_...)

[Div12345]

@ErikBjare If you think the current level won't cause any problems, we could try a deploy and visually see all the problems/errors that are occurring.. @sylvchev is okay with it.

[ErikBjare]

@Div12345 For sure, this PR has been open for too long.

I say go ahead and merge, and we'll fix the remaining issues down the line in follow-up PRs.

[Div12345]

The HTML is available as artifact here

Issues Pending -

• In the Tutorials, only the Getting Started one shows the output.
• In the Examples, Select electrodes and resampling (select_electrodes_resample.py) and MNE Epochs-based piplines (mne_and_scikit_estimators.py) don't show output
• In the Analysis section - meta_analysis.collapse_session_scores(df) doesn't have a description
• In Datasets section - several of the datasets show [1]_ at the end without the link leading anywhere
• In Gallery, two examples show the title as Cross Session SSVEP - May consider changing it a bit
• In the front page, seems like a few formatting errors and the Disclaimer shows Test as Failing

[sylvchev]

I corrected most of the error indicated. The most disturbing problem is that select_electrodes_resample.py, mne_and_scikit_estimators.py and all tutorial_X_...pyare not generated and this is also the case when you build the doc locally. I verified that this is not linked with the Jan's issue #194, neither than with a problem of MOABB caching the results. For example, mne_and_scikit_estimators.py uses the overwrite flag: when executing the code separately, all estimators are recomputed each time.

This is clearly visible in the Sphinx output, all the concerned examples and tutorials have a 0.00 sec allocated time:

computation time summary:
- ../../examples/external/plot_learning_curve_p300_external.py: 50.51 sec 0.0 MB
- ../../examples/plot_learning_curve_p300.py: 36.83 sec 0.0 MB
- ../../examples/plot_within_session_p300.py: 21.25 sec 0.0 MB
- ../../examples/plot_statistical_analysis.py: 20.37 sec 0.0 MB
- ../../examples/plot_filterbank_csp_vs_csp.py: 19.05 sec 0.0 MB
- ../../examples/plot_learning_curve_motor_imagery.py: 16.03 sec 0.0 MB
- ../../examples/plot_cross_session_multiple_datasets.py: 11.53 sec 0.0 MB
- ../../examples/plot_explore_paradigm.py: 11.21 sec 0.0 MB
- ../../examples/plot_cross_session_motor_imagery.py: 6.42 sec 0.0 MB
- ../../tutorials/plot_Getting_Started.py: 5.58 sec 0.0 MB
- ../../examples/plot_cross_session_ssvep.py: 2.03 sec 0.0 MB
- ../../examples/plot_cross_subject_ssvep.py: 1.58 sec 0.0 MB
- ../../examples/mne_and_scikit_estimators.py: 0.00 sec 0.0 MB
- ../../examples/select_electrodes_resample.py: 0.00 sec 0.0 MB
- ../../tutorials/tutorial_1_simple_example_motor_imagery.py: 0.00 sec 0.0 MB
- ../../tutorials/tutorial_2_using_mulitple_datasets.py: 0.00 sec 0.0 MB
- ../../tutorials/tutorial_3_benchmarking_multiple_pipelines.py: 0.00 sec 0.0 MB
- ../../tutorials/tutorial_4_adding_a_dataset.py: 0.00 sec 0.0 MB

@ErikBjare Any ideas why some of these examples and tutorials are skipped by auto_examples and auto_tutorials?
Also, is there some extra step to deploy the documentation on neurotechx website or will it be done when merging this PR?

[sylvchev]

Ok, got it. I RTFM and only files beginning with plot_ are generated. Updating the doc, it should be fully build now.

[sylvchev]

If you are ok with results @Div12345 I'll merge the PR and hopefully we will have a new version available on http://moabb.neurotechx.com/docs/index.html
We could check how to change the site url afterwards