Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add support for skipping steps in Python packages installed as extension + print progress on individual steps for installing Python packages as extensions #2290

Merged

Conversation

Flamefire
Copy link
Contributor

@Flamefire Flamefire commented Dec 21, 2020

(requires easybuilders/easybuild-framework#3561)

Follow-up to easybuilders/easybuild-framework#3524: Currently the test_step is not skipped for PythonPackages in extensions, e.g. TensorFlow for which easybuilders/easybuild-framework#3524 was intended.
This reuses the skip_step machinery from EasyBlock including the status output, which might be useful for things like TensorFlow which have build and test steps taking longer, although this gets quite verbose.

Idea: Don't print a newline, so we have "configuring... building... testing... installing..." there. Maybe we can use \r to replace the line altogether?

Example output:

== fetching files...
== creating build dir, resetting environment...
== unpacking...
== patching...
== preparing...
== configuring...
== building...
== testing...
== installing...
== taking care of extensions...
== installing extension Markdown 3.3.3 (1/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension pyasn1-modules 0.2.8 (2/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension rsa 4.6 (3/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension cachetools 4.2.0 (4/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension google-auth 1.24.0 (5/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension oauthlib 3.1.0 (6/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension requests-oauthlib 1.3.0 (7/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension google-auth-oauthlib 0.4.2 (8/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension Werkzeug 1.0.1 (9/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension absl-py 0.11.0 (10/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension astunparse 1.6.3 (11/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension grpcio 1.32.0 (12/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension tensorboard-plugin-wit 1.7.0 (13/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension tensorboard 2.4.0 (14/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension google-pasta 0.2.0 (15/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension termcolor 1.1.0 (16/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension tensorflow-estimator 2.4.0 (17/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension gast 0.3.3 (18/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension opt-einsum 3.3.0 (19/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension wrapt 1.12.1 (20/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension Keras-Preprocessing 1.1.2 (21/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension dill 0.3.3 (22/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension tblib 1.7.0 (23/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension portpicker 1.3.1 (24/25)...
== 	configuring...
== 	building...
== 	testing...
== 	installing...
== installing extension TensorFlow 2.4.0 (25/25)...
== 	configuring...
== 	building...

@boegel boegel added this to the next release (4.3.3?) milestone Jan 20, 2021
@boegel boegel changed the title Add support for skipping steps in Extension PythonPackages add support for skipping steps in Python packages installed as extension Jan 20, 2021
@boegel boegel changed the title add support for skipping steps in Python packages installed as extension add support for skipping steps in Python packages installed as extension + print progress on individual steps for installing Python packages as extensions Feb 2, 2021
Copy link
Member

@boegel boegel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small but important remark requiring a small PR to framework repo.

I wouldn't try to make the output fancier right now, I have some separate plans for doing that in a larger overhaul anyway.

easybuild/easyblocks/generic/pythonpackage.py Outdated Show resolved Hide resolved
@boegel
Copy link
Member

boegel commented Feb 12, 2021

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS SciPy-bundle-2020.11-foss-2020b.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node3107.skitty.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz, Python 3.6.8
See https://gist.github.com/ee8066ebe2c8e441db7f9e0e249cb95e for a full test report.

@boegel
Copy link
Member

boegel commented Feb 12, 2021

Tested with --skip-test-step, works like a charm. Thanks @Flamefire !

@boegel boegel merged commit e6011be into easybuilders:develop Feb 12, 2021
@Flamefire Flamefire deleted the 20201221091925_new_pr_OeXlUYzDCO branch February 12, 2021 12:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants