Skip to content

Commit

Permalink
[MNT] Improve packaging, managing of dependencies, and managing of ig…
Browse files Browse the repository at this point in the history 
…nored warning/errors (#88)

* migrate to pyproject.toml

* handle version

* drop manifest since tarball does not need any files outside mne_icalabel

* include iclabel network weights

* simpler setup.py

* run draft-pdf only on push with changes to the paper

* add style check in separate workflow through github actions, remove all ignore words

* remove configuration for mypy

* fix typos

* fix typo

* fix flake8

* fix codespell

* don't create file for a single word to ignore

* Revert "don't create file for a single word to ignore"

This reverts commit faee59d.

* try again

* debug

* fix codespell

* match only mne_icalabel for pydocstyle

* minimal mypy configuration

* add # type: ignore

* rework build test and run on windows as well

* move style back to main workflow file

* test without qt for build

* rename build workflow

* try mypy action

* clean-up pytest workflow

* fix typo

* fix typo

* update build ci

* test if ci fails

* Revert "test if ci fails"

This reverts commit b4d57e2.

* try with no underscore?

* ci improvements

* upload to codecov only with one CI

* try again

* simpler

* small improvements

* better

* fix require_version until 1.1 is released

* simplify gui tests

* better error message

* simplify ci for now

* remove some testing req.

* better

* better handling of coverage

* cache the testing dataset

* fix issues

* fix pytest config

* cache also mne-icalabel testing dataset

* comment

* run isort

* better test

* fix import of dataset

* import features in package __init__

* and annotation

* run mne-main only on ubuntu to reduce number of jobs

* remove setup.cfg

* simpler matrix

* retrieve and setup xvfb for ubuntu

* move gui req. to separate key

* fix missed

* fix typo

* try again

* make it executable

* update docbuild

* debug

* try to run Xvfb via action

* test with pg_app

* try to install qt5 dep

* better

* add pytest dep

* remve pytest-xvfb

* cleaning one by one: pytest-qt

* cleaning one-by-one: pydocstyle, pymatreader

* revert pymatreader

* cleaning one-by-one: pytest-harvest

* cleaning one-by-one: pytest-timeout

* set pooch as base dep.

* try without "

* remove empty lines

* move matplotlib to gui dep.

* clean-up comments

* try without Xvfb

* revert, Xvfb is needed as it should be

* try without conftest.py

* disable cache

* fix typos

* Revert "fix typos"

This reverts commit a9f8795.

* Revert "disable cache"

This reverts commit 15e12c4.

* Revert "try without conftest.py"

This reverts commit cd5f502.

* try without unused matplotlib fixtures

* try without markers

* try with empty pytest_configure

* drop conftest.py again

* debug

* add missing __init__.py

* Revert "debug"

This reverts commit 69f7083.

* fix caching of the datasets (stable retrieved 0.132 while main retrieved 0.138)

* Cleanup circle

* Try again

* Add this

* Try again?

* bump mne req to 1.1

* Try again

* use qt6 in CIs

* use -yq flag to uninstall and suppress verbosity

* remove requires_version (fixed in mne 1.1)

* remove 1.0 fixes from topomap

* remove requires_version from gui tests

* try with python3-pyqt6

* typo

* revert to qt5 for tests

* try apt-get

* defaultS -> default ...

* Try circle again

* Try circle again

* Add checkout

* Add checkout

* Fix install for circleci

* Try again

* Try again

* replace scikit-learn doc dep. with ICA key

* minor changes

* fix makefile

* remove non-existent file

* simpler

* use qt5 in doc build

* remove debug

* remove set -e

* split debug information and add caching of the MNE sample dataset

* revert caching of the sample datasets as examples are not always rendered

* does pattern.txt exists?

* nope it does not

* remove setup of xvfb

* Revert "remove setup of xvfb"

This reverts commit 5b9f152.

* what about qt5-default

* try without ffmpeg

* add back set -e to exit upon error and remove mne_icalabel_data directory

* add names to cache operations

* fix linkcheck

* add ls to debug

* more debug

* does circle support xvfb natively?

* what about xvfb-run?

* Revert "what about xvfb-run?"

This reverts commit 08a4d6a.

* Revert "does circle support xvfb natively?"

This reverts commit a9bfc90.

* remove display

* use apt instead of apt-get

* Revert "remove display"

This reverts commit 873507e.

* fix dep. for doc build

* remove forgotten debug lines

* remove unused memory_profiler

* Revert "remove unused memory_profiler"

This reverts commit 1d574fb.

* Update CIs

* Fix template and ci

* Fix circleci

* Fix circleci

* Fix nameing

* Upgrade image

* Upgrade image

* Fix style

* upload artifacts right after building

* try with codecov pinged on python 3.9 build

* install only once

* Clean make

* Modify install page

* Try codecov again

* use latest isort action

* fix readme

* fix workflow name

* better install doc

* add additional skip folders to codespell action

Co-authored-by: Adam Li <adam2392@gmail.com>
  • Loading branch information
@mscheltienne @adam2392
mscheltienne and adam2392 committed Aug 5, 2022
1 parent f740a59 commit a5cdd8f
Show file tree
Hide file tree
Showing 39 changed files with 625 additions and 808 deletions.
197 changes: 101 additions & 96 deletions .circleci/config.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,10 @@ jobs:
- image: cimg/base:stable-20.04
steps:
- restore_cache:
name: Restore .git
keys:
- source-cache-iclabel

- checkout

- run:
name: Complete checkout
command: |
Expand All @@ -19,12 +18,11 @@ jobs:
fi
git remote set-url upstream https://github.com/mne-tools/mne-icalabel.git
git fetch upstream
- save_cache:
name: Save .git
key: source-cache-iclabel
paths:
- ".git"

- run:
name: Check-skip
command: |
Expand All @@ -34,17 +32,23 @@ jobs:
echo "Skip detected, exiting job ${CIRCLE_JOB} for PR ${CIRCLE_PULL_REQUEST}."
circleci-agent step halt;
fi
- add_ssh_keys:
fingerprints:
- "d6:88:6b:a0:80:bf:14:8e:80:2e:ed:11:52:26:37:68"

- run:
name: Merge with upstream
command: |
echo $(git log -1 --pretty=%B) | tee gitlog.txt
echo ${CI_PULL_REQUEST//*pull\//} | tee merge.txt
if [[ $(cat merge.txt) != "" ]]; then
echo "Merging $(cat merge.txt)";
git pull --ff-only upstream "refs/pull/$(cat merge.txt)/merge";
fi
- run:
name: Set BASH_ENV
command: |
set -e
./scripts/setup_xvfb.sh
sudo apt install -qq graphviz optipng python3.8-venv python3-venv libxft2 ffmpeg
curl https://raw.githubusercontent.com/mne-tools/mne-python/main/tools/setup_xvfb.sh -o setup_xvfb.sh
chmod +x setup_xvfb.sh
./setup_xvfb.sh
sudo apt install -qq graphviz optipng python3.8-venv python3-venv libxft2
python3.8 -m venv ~/python_env
echo "set -e" >> $BASH_ENV
echo "export OPENBLAS_NUM_THREADS=4" >> $BASH_ENV
Expand All @@ -56,97 +60,66 @@ jobs:
ln -s ~/python_env/bin/python ~/.local/bin/python
echo "BASH_ENV:"
cat $BASH_ENV
mkdir -p ~/mne_icalabel_data
touch pattern.txt;
# make sure pandoc is on the system
- run: sudo apt update && sudo apt install -y pandoc optipng

- run:
name: Merge with upstream
command: |
echo $(git log -1 --pretty=%B) | tee gitlog.txt
echo ${CI_PULL_REQUEST//*pull\//} | tee merge.txt
if [[ $(cat merge.txt) != "" ]]; then
echo "Merging $(cat merge.txt)";
git pull --ff-only upstream "refs/pull/$(cat merge.txt)/merge";
fi
- run:
name: Install relevant dependencies
command: |
set -e
./scripts/circle_ci_dependencies.sh
# Load pip cache
name: Setup pandoc
command: sudo apt update && sudo apt install -y pandoc optipng
- restore_cache:
name: Restore pip cache
keys:
- pip-cache
- restore_cache:
name: Restore install-bin-cache
keys:
- user-install-bin-cachev1

- user-install-bin-cache
- run:
name: Get Python running and install dependencies
command: |
pip install --progress-bar off .
pip install --upgrade --progress-bar off -r requirements_testing.txt
pip install --upgrade --progress-bar off -r requirements_doc.txt
pip install --upgrade --progress-bar off PyQt5
python -m pip uninstall -yq sphinx-gallery mne-qt-browser
python -m pip install --upgrade --no-deps https://github.com/mne-tools/mne-python/archive/main.zip
# TODO: Revert to upstream/main once https://github.com/mne-tools/mne-qt-browser/pull/105 is merged
python -m pip install --upgrade --progress-bar off https://github.com/mne-tools/mne-qt-browser/zipball/main https://github.com/sphinx-gallery/sphinx-gallery/zipball/master
python -m pip install --progress-bar off --upgrade pip setuptools wheel
python -m pip install --progress-bar off .
python -m pip install --progress-bar off .[doc,gui]
python -m pip uninstall -yq mne mne-qt-browser
python -m pip install --progress-bar off git+https://github.com/mne-tools/mne-python
python -m pip install --progress-bar off git+https://github.com/mne-tools/mne-qt-browser
- save_cache:
name: Save pip cache
key: pip-cache
paths:
- ~/.cache/pip

- save_cache:
key: user-install-bin-cachev1
name: Save install-bin-cache
key: user-install-bin-cache
paths:
- ~/.local/lib/python3.8/site-packages
- ~/.local/bin

# - run:
# name: Check Qt
# command: LD_DEBUG=libs python -c "from PySide6.QtWidgets import QApplication, QWidget; app = QApplication([])"


# Look at what we have and fail early if there is some library conflict
- run:
name: Check pip package versions
command: pip freeze
- run:
name: Display MNE infos
command: QT_DEBUG_PLUGINS=1 mne sys_info -pd
- run:
name: Check installation
command: |
which python
python -c "import mne_icalabel;"
python -c "import mne; mne.sys_info()"
LIBGL_DEBUG=verbose python -c "import matplotlib.pyplot as plt; plt.figure()"
python -c "import mne; mne.set_config('MNE_LOGGING_LEVEL', 'info')"
python -c "import mne; level = mne.get_config('MNE_LOGGING_LEVEL'); assert level.lower() == 'info', repr(level)"
python -c "import numpy; numpy.show_config()"
# Build doc
- run:
name: make html
name: Build documentation
command: |
cd doc;
make html;
# Save the JUnit file
cd doc
make html
# Save the example test results
- store_test_results:
path: doc/_build/test-results

- store_artifacts:
path: doc/_build/test-results
destination: test-results

# Save the SG RST
- store_artifacts:
path: doc/auto_examples.zip
- store_artifacts:
path: doc/generated.zip

# Save the outputs
- store_artifacts:
path: doc/_build/html/
Expand All @@ -161,24 +134,17 @@ jobs:
- html_stable

linkcheck:
# there are a few files excluded from this for expediency, see Makefile
parameters:
scheduled:
type: string
default: "false"
docker:
- image: circleci/python:3.9.2-buster
- image: cimg/python:3.9.2
steps:
- restore_cache:
keys:
- source-cache
- source-cache-iclabel
- checkout
- run:
name: Set BASH_ENV
command: |
set -e
echo "set -e" >> $BASH_ENV
echo "export PATH=~/.local/bin/:$PATH" >> $BASH_ENV
- run:
name: Check-skip
command: |
Expand All @@ -187,13 +153,20 @@ jobs:
echo "Skip detected, exiting job ${CIRCLE_JOB}."
circleci-agent step halt;
fi
- run:
name: Set BASH_ENV
command: |
set -e
echo "set -e" >> $BASH_ENV
echo "export PATH=~/.local/bin/:$PATH" >> $BASH_ENV
- restore_cache:
keys:
- pip-cache
- run:
name: Get Python running
command: |
./scripts/circleci_dependencies.sh
python -m pip install --progress-bar off --upgrade pip setuptools wheel
python -m pip install --progress-bar off .[doc]
- run:
name: Check installation
command: |
Expand All @@ -211,19 +184,25 @@ jobs:
path: doc/_build/linkcheck
destination: linkcheck

doc-deploy:
docker:
- image: cimg/node:lts
deploy:
machine:
image: cimg/node:lts
steps:
- checkout

- attach_workspace:
at: doc/_build

- restore_cache:
keys:
- website-cache

at: /doc/_build
- run:
name: Set BASH_ENV
command: |
set -e
echo "set -e" >> $BASH_ENV
# Don't try to deploy if nothing is there or not on the right branch
- run:
name: Check docs
command: |
if [ ! -f /doc/_build/html/index.html ] && [ ! -f /doc/_build/html_stable/index.html ]; then
echo "No files found to upload (build: ${CIRCLE_BRANCH}).";
circleci-agent step halt;
fi;
- run:
name: Install and configure dependencies
# do not update gh-pages above 3.0.0
Expand All @@ -232,11 +211,9 @@ jobs:
npm install gh-pages@3.0
git config --global user.email "circle@mne.com"
git config --global user.name "Circle Ci"
- add_ssh_keys:
fingerprints:
- "b5:1e:a1:6d:8d:48:f2:8f:dd:bd:2d:66:a9:30:fe:b9"

- run:
# push built doc into the `dev` directory on the `gh-pages` branch
name: Deploy doc to gh-pages branch
Expand All @@ -246,25 +223,53 @@ jobs:
node_modules/gh-pages/bin/gh-pages.js --dotfiles --message "doc updates [skip ci]" --dist doc/_build/html --dest ./dev
else
echo "Deploying stable doc for ${CIRCLE_BRANCH}.";
node_modules/gh-pages/bin/gh-pages.js --dotfiles --message "doc updates [skip ci]" --dist doc/_build/html --dest ./stable
node_modules/gh-pages/bin/gh-pages.js --dotfiles --message "doc updates [skip ci]" --dist doc/_build/html_stable --dest ./stable
fi;
- save_cache:
key: website-cache
paths:
- ~/mne-icalabel

workflows:
default:
jobs:
- build_doc:
name: build_doc
- linkcheck:
name: linkcheck
- doc-deploy:
- deploy:
requires:
- build_doc
filters:
branches:
only:
- main
- maint/0.2

main:
jobs:
- build_doc:
name: build_doc_main
- deploy:
name: deploy_main
requires:
- build_doc_main
triggers:
- schedule:
# "At 00:00" (once a day) should be enough "0 0 * * *",
# But for testing at first, let's do once an hour (6 AM GMT)
cron: "0 6 * * *"
filters:
branches:
only:
- main

weekly:
jobs:
- linkcheck:
name: linkcheck_weekly
scheduled: "true"
triggers:
- schedule:
# "At 00:00 on Sunday" should be often enough
cron: "0 0 * * 0"
filters:
branches:
only:
- main
1 change: 1 addition & 0 deletions .codespellignore
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
raison
26 changes: 26 additions & 0 deletions .flake8
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
[flake8]
max-line-length = 100

ignore =
# these rules don't play well with black
# whitespace before ':'
E203
# line break before binary operator
W503
E241,E305,W504,W605,E731

exclude =
.git
.github
.venv
.mypy_cache
.pytest_cache
.circleci
paper
setup.py
doc

per-file-ignores =
# __init__.py files are allowed to have unused imports
*/__init__.py:F401
*/**/__init__.py:F401
10 changes: 7 additions & 3 deletions .github/PULL_REQUEST_TEMPLATE.md
View file
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
Thanks for contributing. If this is your first time,
make sure to read the [contributing guide.](https://github.com/mne-tools/mne-python/blob/main/CONTRIBUTING.md)
<!--
Thanks for contributing this pull request (PR).
If this is your first time, make sure to read
[CONTRIBUTING.md](https://github.com/mne-tools/mne-python/blob/main/CONTRIBUTING.md)
-->

PR Description
--------------

Describe your PR here
<!--Describe your PR here-->

Merge checklist
---------------
Expand All @@ -16,4 +19,5 @@ Maintainer, please confirm the following before merging:
- [ ] All CIs are happy
- [ ] PR title starts with [MRG]
- [ ] [whats_new.rst](https://github.com/mne-tools/mne-connectivity/blob/main/doc/whats_new.rst) is updated
- [ ] New contributors have been added to [CITATION.cff](https://github.com/mne-tools/mne-icalabel/blob/main/CITATION.cff)
- [ ] PR description includes phrase "closes <#issue-number>"
Loading

0 comments on commit a5cdd8f

Please sign in to comment.