PyBuilder becomes stuck on unit test execution (and other test failures)

[hthompson6]

Description

I've been trying to go through the installation steps, but encountered the issues listed below. I didn't see a Python version specified, so I tried a couple of them. Though, I had seen that 3.6 was used for unit tests in another issue.

Note: I was able to install and execute the tutorial code successfully on python3.8. Only the final PyBuilder unit test step was problematic.

Environment

• WSL2 running Ubuntu 20.04 on Windows 10

Manual Testing

Attempted Installation Using Python3.6

Fails due to future annotations not being available before python3.7

Steps

cd lava
virtualenv --python=/usr/bin/python3.6 ~/env_lava_36
source ~/env_lava_36/bin/activate
pip install -U pip
pip install -r build-requirements.txt
pip install -r requirements.txt
pip install .
export PYTHONPATH=$(pwd)/src
pyb -E unit

Output

PyBuilder version 0.13.3
Build started at 2021-11-30 17:47:44
------------------------------------------------------------
[INFO]  Installing or updating plugin "pypi:pybuilder_bandit, module name 'pybuilder_bandit'"
[INFO]  Processing plugin packages 'pybuilder_bandit' to be installed with {}
[INFO]  Activated environments: unit
[INFO]  Building lava-nc version 0.2.0
[INFO]  Executing build in /home/flynnto/self_study/lava
[INFO]  Going to execute tasks: analyze, publish
[INFO]  Processing plugin packages 'coverage~=5.2' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'flake8~=3.7' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'pypandoc~=1.4' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'setuptools>=38.6.0' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'sphinx_rtd_theme' to be installed with {}
[INFO]  Processing plugin packages 'sphinx_tabs' to be installed with {}
[INFO]  Processing plugin packages 'twine>=1.15.0' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'unittest-xml-reporting~=3.0.4' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'wheel>=0.34.0' to be installed with {'upgrade': True}
[INFO]  Creating target 'build' VEnv in '/home/flynnto/self_study/lava/target/venv/build/cpython-3.6.15.final.0'
[INFO]  Processing dependency packages 'requirements.txt' to be installed with {}
[INFO]  Creating target 'test' VEnv in '/home/flynnto/self_study/lava/target/venv/test/cpython-3.6.15.final.0'
[INFO]  Processing dependency packages 'requirements.txt' to be installed with {}
[INFO]  Requested coverage for tasks: pybuilder.plugins.python.unittest_plugin:run_unit_tests
[INFO]  Running unit tests
[INFO]  Executing unit tests from Python modules in /home/flynnto/self_study/lava/tests/lava
------------------------------------------------------------
BUILD FAILED - SyntaxError: future feature annotations is not defined (lava/magma/core/process/process.py:9)
------------------------------------------------------------
Build finished at 2021-11-30 17:47:54
Build took 10 seconds (10169 ms)

Attempted Installation Using Python3.7

Fails due to f-string formatting

Steps

cd lava
virtualenv --python=/usr/bin/python3.7 ~/env_lava_37
source ~/env_lava_37/bin/activate
pip install -U pip
pip install -r build-requirements.txt
pip install -r requirements.txt
pip install .
export PYTHONPATH=$(pwd)/src
pyb -E unit

Output

PyBuilder version 0.13.3
Build started at 2021-11-30 17:50:00
------------------------------------------------------------
[INFO]  Installing or updating plugin "pypi:pybuilder_bandit, module name 'pybuilder_bandit'"
[INFO]  Processing plugin packages 'pybuilder_bandit' to be installed with {}
[INFO]  Activated environments: unit
[INFO]  Building lava-nc version 0.2.0
[INFO]  Executing build in /home/flynnto/self_study/lava
[INFO]  Going to execute tasks: analyze, publish
[INFO]  Processing plugin packages 'coverage~=5.2' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'flake8~=3.7' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'pypandoc~=1.4' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'setuptools>=38.6.0' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'sphinx_rtd_theme' to be installed with {}
[INFO]  Processing plugin packages 'sphinx_tabs' to be installed with {}
[INFO]  Processing plugin packages 'twine>=1.15.0' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'unittest-xml-reporting~=3.0.4' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'wheel>=0.34.0' to be installed with {'upgrade': True}
[INFO]  Creating target 'build' VEnv in '/home/flynnto/self_study/lava/target/venv/build/cpython-3.7.12.final.0'
[INFO]  Processing dependency packages 'requirements.txt' to be installed with {}
[INFO]  Creating target 'test' VEnv in '/home/flynnto/self_study/lava/target/venv/test/cpython-3.7.12.final.0'
[INFO]  Processing dependency packages 'requirements.txt' to be installed with {}
[INFO]  Requested coverage for tasks: pybuilder.plugins.python.unittest_plugin:run_unit_tests
[INFO]  Running unit tests
[INFO]  Executing unit tests from Python modules in /home/flynnto/self_study/lava/tests/lava
------------------------------------------------------------
BUILD FAILED - SyntaxError: invalid syntax (<fstring>, line 1) (lava/magma/runtime/runtime.py:10)
------------------------------------------------------------
Build finished at 2021-11-30 17:50:08
Build took 8 seconds (8513 ms)

Attempted Installation Using Python3.8

Stuck for >1 hr

Steps

cd lava
virtualenv --python=/usr/bin/python3.8 ~/env_lava_38
source ~/env_lava_38/bin/activate
pip install -U pip
pip install -r build-requirements.txt
pip install -r requirements.txt
pip install .
export PYTHONPATH=$(pwd)/src
pyb -E unit

Output

PyBuilder version 0.13.3
Build started at 2021-11-30 17:52:57
------------------------------------------------------------
[INFO]  Installing or updating plugin "pypi:pybuilder_bandit, module name 'pybuilder_bandit'"
[INFO]  Processing plugin packages 'pybuilder_bandit' to be installed with {}
[INFO]  Activated environments: unit
[INFO]  Building lava-nc version 0.2.0
[INFO]  Executing build in /home/flynnto/self_study/lava
[INFO]  Going to execute tasks: analyze, publish
[INFO]  Processing plugin packages 'coverage~=5.2' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'flake8~=3.7' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'pypandoc~=1.4' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'setuptools>=38.6.0' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'sphinx_rtd_theme' to be installed with {}
[INFO]  Processing plugin packages 'sphinx_tabs' to be installed with {}
[INFO]  Processing plugin packages 'twine>=1.15.0' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'unittest-xml-reporting~=3.0.4' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'wheel>=0.34.0' to be installed with {'upgrade': True}
[INFO]  Creating target 'build' VEnv in '/home/flynnto/self_study/lava/target/venv/build/cpython-3.8.10.final.0'
[INFO]  Processing dependency packages 'requirements.txt' to be installed with {}
[INFO]  Creating target 'test' VEnv in '/home/flynnto/self_study/lava/target/venv/test/cpython-3.8.10.final.0'
[INFO]  Processing dependency packages 'requirements.txt' to be installed with {}
[INFO]  Requested coverage for tasks: pybuilder.plugins.python.unittest_plugin:run_unit_tests
[INFO]  Running unit tests
[INFO]  Executing unit tests from Python modules in /home/flynnto/self_study/lava/tests/lava
Runtime not started yet.
[Loihi1SimCfg]: Using the first PyProcessModel PyLifModelFloat available for Process Process_144::LIF.
[Loihi1SimCfg]: Using the first PyProcessModel PyLifModelFloat available for Process Process_147::LIF.

Attempted Installation Using Python3.9

Stuck for >1 hr

Steps

cd lava
virtualenv --python=/usr/bin/python3.9 ~/env_lava_39
source ~/env_lava_39/bin/activate
pip install -U pip
pip install -r build-requirements.txt
pip install -r requirements.txt
pip install .
export PYTHONPATH=$(pwd)/src
pyb -E unit

Output

PyBuilder version 0.13.3
Build started at 2021-11-30 17:55:40
------------------------------------------------------------
[INFO]  Installing or updating plugin "pypi:pybuilder_bandit, module name 'pybuilder_bandit'"
[INFO]  Processing plugin packages 'pybuilder_bandit' to be installed with {}
[INFO]  Activated environments: unit
[INFO]  Building lava-nc version 0.2.0
[INFO]  Executing build in /home/flynnto/self_study/lava
[INFO]  Going to execute tasks: analyze, publish
[INFO]  Processing plugin packages 'coverage~=5.2' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'flake8~=3.7' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'pypandoc~=1.4' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'setuptools>=38.6.0' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'sphinx_rtd_theme' to be installed with {}
[INFO]  Processing plugin packages 'sphinx_tabs' to be installed with {}
[INFO]  Processing plugin packages 'twine>=1.15.0' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'unittest-xml-reporting~=3.0.4' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'wheel>=0.34.0' to be installed with {'upgrade': True}
[INFO]  Creating target 'build' VEnv in '/home/flynnto/self_study/lava/target/venv/build/cpython-3.9.5.final.0'
[INFO]  Processing dependency packages 'requirements.txt' to be installed with {}
[INFO]  Creating target 'test' VEnv in '/home/flynnto/self_study/lava/target/venv/test/cpython-3.9.5.final.0'
[INFO]  Processing dependency packages 'requirements.txt' to be installed with {}
[INFO]  Requested coverage for tasks: pybuilder.plugins.python.unittest_plugin:run_unit_tests
[INFO]  Running unit tests
[INFO]  Executing unit tests from Python modules in /home/flynnto/self_study/lava/tests/lava
Runtime not started yet.
[Loihi1SimCfg]: Using the first PyProcessModel PyLifModelFloat available for Process Process_144::LIF.
[Loihi1SimCfg]: Using the first PyProcessModel PyLifModelFloat available for Process Process_147::LIF.

[harryliu-intel]

The minimum Python version is 3.8, and we will document it more prominently per issue #91. I was able the replicate the hanging behavior and I think it has the same root cause as issue #71, with the same workaround: could you try to increase the limit on the number of file descriptors (ulimit -n 2048) before running the unit tests? That allowed "pyb -E unit" to complete successfully for me.

[hthompson6]

Increasing the file descriptor limit resolved the issue. Thank you!