Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Speeding up conda solver time by removing pymapd and pinning some packages #2193

Merged
merged 11 commits into from
Apr 23, 2020

Conversation

datapythonista
Copy link
Contributor

xref #2181

Running conda env create --name ibis-dev --file ci/requirements-3.7-dev.yml is currently taking 36 minutes (for the conda solver). This causes the builds to reach the 1 hour limit and fail (the docs, benchmarks and conda build builds).

Pinning pymapd reduces the time to to 20 minutes (from the 36). We still need a better solution (20 minutes for the solver is crazy), but at least this should fix the CI in an easy way for now.

@jreback
Copy link
Contributor

jreback commented Apr 23, 2020

let’s remove pymapd entirely for now

this broke a lot of things

@datapythonista
Copy link
Contributor Author

Removed the dependency here, so we can see the change in the solver speed in the CI for now. I guess we need to disable the backend from the tests too. Anything else?

@datapythonista datapythonista changed the title CI: Speeding up conda solver time by pinning pymapd CI: Speeding up conda solver time by removing pymapd and pinning some packages Apr 23, 2020
@datapythonista
Copy link
Contributor Author

@jreback this solves the main issue with the conda solver being too slow. The remaining errors are the ones caused by not having enough space in disk. And seems like the LinuxBuildConda has another set of conda dependencies that are also too slow. I'll have a look to that in a separate PR.

If we can get this merged, that would be very helpful. So, I can finish the disk space problems without conda being too slow. Thanks!

@jreback jreback added the ci Continuous Integration issues or PRs label Apr 23, 2020
@jreback jreback added this to the Next Bugfix Release milestone Apr 23, 2020
@jreback jreback merged commit 3d9756c into ibis-project:master Apr 23, 2020
@jreback
Copy link
Contributor

jreback commented Apr 23, 2020

thanks @datapythonista IIRC @xmnlab has some patches for the image size. though maybe we just want to split the testing in 2

@datapythonista
Copy link
Contributor Author

I'm testing on splitting the builds in two in #2194, and seems quite simple. Will re-run now that PR with this fixed, and see how it looks. I think just two are missing if we go for the splitting approach:

  • See what to do with doctests, which some will fail if the backend is not set up (can't skip doctests with the marker as the rest)
  • The long conda solve in the LinuxBuildConda build

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci Continuous Integration issues or PRs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants