Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache SLAM Dataset for CI #279

Open
mhuzai opened this issue Apr 22, 2021 · 4 comments
Open

Cache SLAM Dataset for CI #279

mhuzai opened this issue Apr 22, 2021 · 4 comments
Labels
infrastructure Build and deployment infrastructure
Milestone

Comments

@mhuzai
Copy link
Member

mhuzai commented Apr 22, 2021

EuRoC V1_02_Medium gets downloaded and unzipped in every single test run. This not only makes tests run longer, but probably also creates (a lot of) unnecessary traffic for the ETH server. And this is about to get worse because we just added a bunch of new CI pipelines. So, we need to find a way to download and unzip this dataset once, cache it, and then use the cache in the test runs.

@mhuzai mhuzai added the infrastructure Build and deployment infrastructure label Apr 22, 2021
@mhuzai mhuzai added this to the 2.x milestone Apr 22, 2021
@mhuzai mhuzai added this to Issue Backlog in ILLIXR improvements via automation Apr 22, 2021
@charmoniumQ
Copy link
Member

Clarification: Every single test run in CI. Local tests (with a persistent disk) do not suffer this problem.

We need to use GitHub actions caching here.

@e3m3
Copy link
Contributor

e3m3 commented Apr 22, 2021

And this is about to get worse because we just added a bunch of new CI pipelines.

Each CI pipeline (flow in ILLIXR/configs/ci/flows introduced in #270) is still embedded within one call to Runner at the moment, so the download is amortized over each pipeline. However, ideally each pipeline will be processed independently by CI to achieve parallelism, necessitating caching.

@charmoniumQ
Copy link
Member

charmoniumQ commented Apr 23, 2021

The dataset gets downloaded to .cache. Saving it also saves the external plugins (Kimera).

We should also consider moving .deps into .cache. Same with the conda env.

@e3m3
Copy link
Contributor

e3m3 commented Apr 26, 2021

We should also consider moving .deps into .cache. Same with the conda env.

These are now in .cache in #264.

@e3m3 e3m3 linked a pull request May 14, 2021 that will close this issue
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infrastructure Build and deployment infrastructure
Projects
ILLIXR improvements
  
Issue Backlog
Development

Successfully merging a pull request may close this issue.

3 participants