New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Caching things in Travis build #1394

Closed
richardjgowers opened this Issue Jun 12, 2017 · 9 comments

Comments

Projects
None yet
4 participants
@richardjgowers
Member

richardjgowers commented Jun 12, 2017

As originally spotted by @utkbansal, installing our dependencies on travis can take 6-7 minutes, we could use the caching facility on travis to reduce this.

Our build is a little complicated as we're using conda, but it looks like scikit-learn have done this:
scikit-learn/scikit-learn#5455

@utkbansal is this something that you could look at?

@utkbansal

This comment has been minimized.

Show comment
Hide comment
@utkbansal

utkbansal Jun 14, 2017

Member

I'll try to read more about this and get started on the coming weekend.

Member

utkbansal commented Jun 14, 2017

I'll try to read more about this and get started on the coming weekend.

@utkbansal

This comment has been minimized.

Show comment
Hide comment
@utkbansal

utkbansal Jun 15, 2017

Member

@richardjgowers Looks like you will have to turn on this feature in TravisCI, it's not available by default.

To use the caching feature, in your repository settings, set Build branch updates to ON.

Member

utkbansal commented Jun 15, 2017

@richardjgowers Looks like you will have to turn on this feature in TravisCI, it's not available by default.

To use the caching feature, in your repository settings, set Build branch updates to ON.

@kain88-de

This comment has been minimized.

Show comment
Hide comment
@kain88-de

kain88-de Jun 15, 2017

Member

Caches should be activated.

Member

kain88-de commented Jun 15, 2017

Caches should be activated.

@orbeckst

This comment has been minimized.

Show comment
Hide comment
@orbeckst

orbeckst Jun 15, 2017

Member

I think we need to

  • cache pip (basically, ~/.pip/cache); I think we also need to make sure that we use pip install --upgrade so that we do pull in new versions of dependencies and not inadvertently keep stale packages around
  • cache conda

Perhaps something like

cache: 
   pip: true
   directories:
       - $HOME/miniconda

will work?

Member

orbeckst commented Jun 15, 2017

I think we need to

  • cache pip (basically, ~/.pip/cache); I think we also need to make sure that we use pip install --upgrade so that we do pull in new versions of dependencies and not inadvertently keep stale packages around
  • cache conda

Perhaps something like

cache: 
   pip: true
   directories:
       - $HOME/miniconda

will work?

@utkbansal utkbansal referenced this issue Jun 16, 2017

Closed

Add caching in TravisCI #1405

0 of 4 tasks complete
@utkbansal

This comment has been minimized.

Show comment
Hide comment
@utkbansal

utkbansal Jun 16, 2017

Member

@kain88-de @orbeckst @richardjgowers @jbarnoud Why are we dependent on astropy's ci helpers? Why don't we have our own?

We might need to modify them to work with caching.

Member

utkbansal commented Jun 16, 2017

@kain88-de @orbeckst @richardjgowers @jbarnoud Why are we dependent on astropy's ci helpers? Why don't we have our own?

We might need to modify them to work with caching.

@kain88-de

This comment has been minimized.

Show comment
Hide comment
@kain88-de

kain88-de Jun 16, 2017

Member

They are very responsive and so far did everything we needed them to do. They are also used by a lot of other projects so we can assume they do stuff correct. If you have problems with the script you can run from a fork temporarily and then we fix it upstream. They might be also interested in the caching.

Member

kain88-de commented Jun 16, 2017

They are very responsive and so far did everything we needed them to do. They are also used by a lot of other projects so we can assume they do stuff correct. If you have problems with the script you can run from a fork temporarily and then we fix it upstream. They might be also interested in the caching.

@orbeckst

This comment has been minimized.

Show comment
Hide comment
@orbeckst

orbeckst Jun 20, 2017

Member

Sort of related to the problem of keeping our run time on travis under 50 mins: We could split the tests in analysis and core using the build matrix (which was suggested in #286 a long time ago). However, the problem has been to maintain coverage. Assuming that #1369 works and we can merge the coverage reports we could now make use of Travis' new feature Build Stages: As I understand it, we can run our matrix as one stage and then run another stage that combines the coverage reports (if the first stage is successful). We could use a GitHub repo or an S3 bucket as temporary storage for coverage reports; I don't know if Travis allows us to keep build artifacts from earlier stages.

Member

orbeckst commented Jun 20, 2017

Sort of related to the problem of keeping our run time on travis under 50 mins: We could split the tests in analysis and core using the build matrix (which was suggested in #286 a long time ago). However, the problem has been to maintain coverage. Assuming that #1369 works and we can merge the coverage reports we could now make use of Travis' new feature Build Stages: As I understand it, we can run our matrix as one stage and then run another stage that combines the coverage reports (if the first stage is successful). We could use a GitHub repo or an S3 bucket as temporary storage for coverage reports; I don't know if Travis allows us to keep build artifacts from earlier stages.

@kain88-de

This comment has been minimized.

Show comment
Hide comment
@kain88-de

kain88-de Aug 5, 2017

Member

One thing. Since we now run different python versions on travis the caching from help us. Dependencies have to be downloaded again for each new python version. Right now I only see general improvements in conda to reduce our installation times. Or alternatively reducing the number of dependencies we have.

Member

kain88-de commented Aug 5, 2017

One thing. Since we now run different python versions on travis the caching from help us. Dependencies have to be downloaded again for each new python version. Right now I only see general improvements in conda to reduce our installation times. Or alternatively reducing the number of dependencies we have.

@orbeckst

This comment has been minimized.

Show comment
Hide comment
@orbeckst

orbeckst Dec 15, 2017

Member

This looks as if we decided not to pursue caching at this time.

I will close and if anyone is interested, please reopen.

Member

orbeckst commented Dec 15, 2017

This looks as if we decided not to pursue caching at this time.

I will close and if anyone is interested, please reopen.

@orbeckst orbeckst closed this Dec 15, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment