Dagster should be distributed through conda

[mgasner]

It should be possible to run conda install dagster.

[mgasner]

It looks like to make this happen we may also need to build and distribute reloader and toposort for conda. Going to see how heavy a responsibility this is.

[mgasner]

https://conda.io/docs/user-guide/tutorials/build-pkgs-skeleton.html
https://conda.io/docs/user-guide/tasks/build-packages/install-conda-build.html#install-conda-build
https://stackoverflow.com/questions/50883081/conda-build-unsatisfiable-dependencies-for-platform-osx-64-torchversion
https://conda.io/docs/user-guide/tasks/build-packages/index.html

[xhochy]

PR for conda-forge: conda-forge/staged-recipes#10285
PR for correcting the license information to make conda-forge packaging simpler: #1938

[mgasner]

@xhochy thank you for taking these steps -- can you provide any guidance on maintaining the recipes in conda-forge/staged-recipes@a861d93? i'm conscious that these details, esp. requirements, are going to change and would like to make sure that as they do dagster remains available on conda.

[xhochy]

Once conda-forge/staged-recipes#10285 is merged, a repo per package will be created. Once a new Dagster version is released, a bot will issue PRs against all repos with the new version and the new sha256 for the sdists. We then need to manually update the runtime requirements and merge all PRs. The bot is currently not able to update the dependencies as well, it might be able to do so in future.

[xhochy]

Feel free to reach out on the to-be-created http://github.com/conda-forge/dagster-feedstock repo if you also want to get maintainer status (i.e. merge rights) on the feedstocks.

[mgasner]

Thanks @xhochy, that's very helpful. I assume that bot will not issue PRs for prereleases, and will definitely reach out for maintainer status on the feedstocks.

[mgasner]

Closing this as we can now be installed with conda, I'll follow up separately re the above.

[natekupp]

Reopening this issue - it looks like our conda-forge releases on https://github.com/conda-forge/dagster-feedstock are currently on 0.7.3, about 2 months old. We should document updating conda as part of our release process

[sephib]

Trying to run
conda install -c conda-forge -y dagster-pandas dagster-dask dagster-pyspark dagster-ge dagster=0.7.11 dagit

Results with conflicts

Package dagster conflicts for: dagster=0.7.11 dagster-ge -> dagster[version='0.6.9.*|0.7.0.*|0.7.1.*|0.7.2.*|0.7.3.*'] dagit -> dagster[version='0.6.5.*|0.6.6.*|0.6.7.*|0.6.7.post0.*|0.6.9.*|0.7.0.*|0.7.1.*|0.7.11.post0.*|0.7.3.*|0.7.2.*'] dagster-pyspark -> dagster[version='0.6.9.*|0.7.1.*|0.7.11.post0.*|0.7.3.*'] dagster-pandas -> dagster[version='0.6.9.*|0.7.0.*|0.7.1.*|0.7.11.post0.*|0.7.3.*|0.7.2.*'] dagster-dask -> dagster[version='0.6.9.*|0.7.0.*|0.7.1.*|0.7.11.post0.*|0.7.3.*|0.7.2.*']

Package ca-certificates conflicts for:
python=3.7 -> openssl[version='>=1.1.1e,<1.1.2a'] -> ca-certificates
dagster-ge -> python -> ca-certificates
dagster-pandas -> python -> ca-certificates
dagster-pyspark -> python[version='<3.8.0a0'] -> ca-certificates
dagit -> python -> ca-certificates

Package typing conflicts for:
dagster-pandas -> dagster=0.7.3 -> typing
dagster-pyspark -> dagster=0.7.3 -> typing
dagit -> dagster=0.7.3 -> typing
dagster-dask -> dagster=0.7.3 -> typing
dagster-ge -> dagster=0.7.3 -> typing

Package argh conflicts for:
dagit -> watchdog[version='>=0.8.3'] -> argh[version='>=0.24.1']
dagster-ge -> great-expectations[version='<=0.7.6'] -> argh[version='>=0.26.2']
dagster=0.7.11 -> watchdog[version='>=0.8.3'] -> argh[version='>=0.24.1']

Package pandas conflicts for:
dagster-dask -> bokeh -> pandas[version='>=0.19.0|>=0.21.0|>=0.23.0']
dagster-pyspark -> pyspark -> pandas[version='>=0.13.0|>=0.19.2']
dagster-ge -> great-expectations[version='<=0.7.6'] -> pandas[version='>=0.20.3|>=0.22.0']
dagster-pandas -> pandas

Package setuptools conflicts for:
dagster=0.7.11 -> setuptools
dagster-pyspark -> dagster=0.7.11.post0 -> setuptools
python=3.7 -> pip -> setuptools
dagster-dask -> dagster=0.7.11.post0 -> setuptools
dagit -> dagster=0.7.11.post0 -> setuptools
dagster-pandas -> dagster=0.7.11.post0 -> setuptools

Package tqdm conflicts for:
dagster-dask -> dagster=0.7.11.post0 -> tqdm
dagster=0.7.11 -> tqdm
dagit -> dagster=0.7.11.post0 -> tqdm
dagster-pandas -> dagster=0.7.11.post0 -> tqdm
dagster-pyspark -> dagster=0.7.11.post0 -> tqdm

Package dagster-graphql conflicts for:
dagit -> dagster-graphql[version='0.6.9.|0.7.0.|0.7.1.|0.7.11.post0.|0.7.3.|0.7.2.']
dagster-dask -> dagster-graphql[version='0.6.9.|0.7.0.|0.7.1.|0.7.11.post0.|0.7.3.|0.7.2.']

Package backports.functools_lru_cache conflicts for:
dagster-ge -> great-expectations[version='<=0.7.6'] -> backports.functools_lru_cache[version='>=1.5']
dagster-pandas -> matplotlib-base -> backports.functools_lru_cache

Package gevent-websocket conflicts for:
dagit -> gevent-websocket[version='>=0.10.1']
dagit -> flask-sockets[version='>=0.2.1'] -> gevent-websocket

Package tornado conflicts for:
dagster-pandas -> matplotlib-base -> tornado
dagster-dask -> bokeh -> tornado[version='>=4.3,<4.5|>=4.3|>=5|>=6.0.3']

Package greenlet conflicts for:
dagster=0.7.11 -> gevent -> greenlet[version='>=0.4.10|>=0.4.13|>=0.4.14']
dagit -> gevent -> greenlet[version='>=0.4.10|>=0.4.13|>=0.4.14']

[mgasner]

@natekupp when we delete modules, we mess up the whole conda pipeline - see conda-forge/dagster-feedstock#24 -- this should now be resolved

[mgasner]

@sephib this should be resolved as of 0.12

[sephib]

Thx