try pytorch again

[ngam]

fixes #358

I will also add a crucial test for jaxlib here, forgot to add it in #364

[github-actions]

👈 Try on Mybinder.org!
👈 Try on Pangeo GCP Binder!
👈 Try on Pangeo AWS Binder!

[pangeo-bot]

/condalock
Automatically locking new conda environment, building, and testing images...

[ngam]

/condalock again

[rabernat]

Thanks so much for working on this. LGTM, but I don't fully understand the details.

[ngam]

Thanks so much for working on this. LGTM, but I don't fully understand the details.

We actually have two potential silent errors and I am trying to make sure we don't face them here in the containers. Our conda-forge tensorflow build is usually solid, and is designed to take precedence, so we usually run into problems with jaxlib (especially when in the same env as tensorflow) along the lines:

• a too strict, misplaced dependency on abseil that slips through, e.g. tracking here relax pin on jaxlib when tf--jax issues are resolved #368, should be fixed in a few hours (nothing is wrong per se, it just needs a specific abseil that tensorflow doesn't accept yet, so it ends up silently breaking things)
• both tensorflow and jaxlib utilize protobuf, grpc, and abseil, but they reshare some definitions, and so if they both use the shared library, they end up conflicting, e.g. see this issue about protobuf jaxlib 0.3.0 conda-forge package is broken conda-forge/jaxlib-feedstock#89 but similar issues arise due to abseil, etc. --- this issue often only manifests after one imports both in the same session, but stays silent otherwise

For use, people will likely need a separate dataloader/datapipeline to use with jax anyway, so it makes sense to make jax and tensorflow live together in harmony --- we likely could and should add pytorch to the mix soon! I am not certain about bandwidth/storage limitation but having a super ML container (containing all platforms a la kaggle or colab) may be a good idea, albeit heavy to store and heavy to pull, etc.

[ngam]

/condalock

[ngam]

@scottyhq is this stuck now or can you resolve the conflicts without going file by file? Maybe best to open a new PR? (Edit: trying to see if the smart bots --- condalock --- can unlock it!)

At any rate, you can safely unpin pytorch like I did here and things should be good to go. I will close this PR this weekend if no further activity (I like to keep open only issues and PRs that are not moot, actually following @rabernat's blog post about using github issues/PRs as a calendar/todo list!)

[ngam]

/condalock

[scottyhq]

/condalock

[scottyhq]

Edit: trying to see if the smart bots --- condalock --- can unlock it!

Unfortunately the bots are not currently smart enough to resolve lockfiles when one active PR is merged before another. It's really a one PR at a time setup right now, which is fine most of the time :)

Thanks again @ngam for all your hard work on the ML library dependencies and packaging