Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only download Mathematica/Wolfram Engine once per workflow #579

Open
maxitg opened this issue Dec 17, 2020 · 2 comments
Open

Only download Mathematica/Wolfram Engine once per workflow #579

maxitg opened this issue Dec 17, 2020 · 2 comments
Labels
infrastructure Has to do with changes to the development process, e.g., build scripts, CI, testing utilities optimization Does not change functionality but makes code faster

Comments

@maxitg
Copy link
Owner

maxitg commented Dec 17, 2020

The problem

Due to tests parallelization #578, we are now downloading the docker image with Mathematica 4 times, which is redundant. We should figure out how to only download it once thus saving 3 minutes' worth of credits. If we download it prior to starting the testing job, we will also reduce the workflow duration by 1 minute.

Possible solution

  1. Take WL out of the Docker image and put it into the Workspace for all nodes to use (tricky because it needs to stay activated).
  2. We can then add another wolfram-engine-download job which is going to put it into the workspace.
  3. The build/test jobs will then just need to put symbolic links/copy relevant files which should be much faster.

Alternative solutions

Circle CI has docker layer caching which might be useful here, however, it is not available on the free plan, and even if it was, it's quite expensive costing more credits than the entire WL testing job.

@maxitg maxitg added optimization Does not change functionality but makes code faster infrastructure Has to do with changes to the development process, e.g., build scripts, CI, testing utilities labels Dec 17, 2020
@aokellermann
Copy link
Collaborator

Documentation for attach_workspace states that:

The full contents of the workspace are downloaded and copied into the directory the workspace is being attached at.

Therefore, if we persist the wolfram installation and attach on our four parallel jobs, it will download the wolfram installation on each one. DLC looks like it could be promising though.

@maxitg
Copy link
Owner Author

maxitg commented Jan 14, 2021

We should even be able to cache it between builds similar to how you cache Python modules.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infrastructure Has to do with changes to the development process, e.g., build scripts, CI, testing utilities optimization Does not change functionality but makes code faster
Projects
None yet
Development

No branches or pull requests

2 participants