Why are PyPI and conda-forge versions for dagster-postgres different?

[zaneselvans]

We are using conda-lock to create a reproducible environment that we run Dagster in. It reads dependencies and version constraints from pyproject.toml, solves the environment using mamba, and produces a complete environment specification in a lockfile. We're in the process of switching to using Postgres instead of SQLite for the EventLogs, since SQLite has been locking up on us due to concurrency. However, when trying to add the new dependency I discovered that the PyPI and conda-forge versioning for the dagster-postgres package seem to be completely unrelated to each other.

• On PyPI the current version is 0.21.6
• On conda-forge the current version is 1.5.5

Is it supposed to be this way? Typically conda packages track the same versions as their PyPI counterparts, and dagster-postgres seems to be an official package. It looks like the 0.21.6 version is probably the "libs" version, while 1.5.5 is the overall Dagster version.

Having these two versioning schemes be out of sync makes it difficult to specify the environment in a straightforward way, because the tools parsing pyproject.toml could ultimately seek the package either from PyPI or conda-forge.

I notice that this seems to be a common, but not universal, situation, across many of the dagster-* packages. Searching conda-forge packages it looks like everything has a most recent version of 1.5.5:

mamba search 'dagster-*==1.5.5'

Loading channels: done
# Name                       Version           Build  Channel
dagster-airbyte                1.5.5    pyhd8ed1ab_0  conda-forge
dagster-airflow                1.5.5    pyhd8ed1ab_0  conda-forge
dagster-aws                    1.5.5    pyhd8ed1ab_0  conda-forge
dagster-celery                 1.5.5    pyhd8ed1ab_0  conda-forge
dagster-celery-docker           1.5.5    pyhd8ed1ab_0  conda-forge
dagster-celery-k8s             1.5.5    pyhd8ed1ab_0  conda-forge
dagster-census                 1.5.5    pyhd8ed1ab_0  conda-forge
dagster-dask                   1.5.5    pyhd8ed1ab_0  conda-forge
dagster-datadog                1.5.5    pyhd8ed1ab_0  conda-forge
dagster-dbt                    1.5.5    pyhd8ed1ab_0  conda-forge
dagster-docker                 1.5.5    pyhd8ed1ab_0  conda-forge
dagster-duckdb                 1.5.5    pyhd8ed1ab_0  conda-forge
dagster-duckdb-pandas           1.5.5    pyhd8ed1ab_0  conda-forge
dagster-duckdb-polars           1.5.5    pyhd8ed1ab_0  conda-forge
dagster-duckdb-pyspark           1.5.5    pyhd8ed1ab_0  conda-forge
dagster-fivetran               1.5.5    pyhd8ed1ab_0  conda-forge
dagster-gcp                    1.5.5    pyhd8ed1ab_0  conda-forge
dagster-gcp-pandas             1.5.5    pyhd8ed1ab_0  conda-forge
dagster-gcp-pyspark            1.5.5    pyhd8ed1ab_0  conda-forge
dagster-ge                     1.5.5    pyhd8ed1ab_0  conda-forge
dagster-github                 1.5.5    pyhd8ed1ab_0  conda-forge
dagster-graphql                1.5.5    pyhd8ed1ab_0  conda-forge
dagster-k8s                    1.5.5    pyhd8ed1ab_0  conda-forge
dagster-managed-elements           1.5.5    pyhd8ed1ab_0  conda-forge
dagster-mlflow                 1.5.5    pyhd8ed1ab_0  conda-forge
dagster-msteams                1.5.5    pyhd8ed1ab_0  conda-forge
dagster-mysql                  1.5.5    pyhd8ed1ab_0  conda-forge
dagster-pagerduty              1.5.5    pyhd8ed1ab_0  conda-forge
dagster-pandas                 1.5.5    pyhd8ed1ab_0  conda-forge
dagster-pandera                1.5.5    pyhd8ed1ab_0  conda-forge
dagster-papertrail             1.5.5    pyhd8ed1ab_0  conda-forge
dagster-pipes                  1.5.5    pyhd8ed1ab_0  conda-forge
dagster-postgres               1.5.5    pyhd8ed1ab_0  conda-forge
dagster-prometheus             1.5.5    pyhd8ed1ab_0  conda-forge
dagster-pyspark                1.5.5    pyhd8ed1ab_0  conda-forge
dagster-shell                  1.5.5    pyhd8ed1ab_0  conda-forge
dagster-slack                  1.5.5    pyhd8ed1ab_0  conda-forge
dagster-snowflake              1.5.5    pyhd8ed1ab_0  conda-forge
dagster-snowflake-pandas           1.5.5    pyhd8ed1ab_0  conda-forge
dagster-snowflake-pyspark           1.5.5    pyhd8ed1ab_0  conda-forge
dagster-spark                  1.5.5    pyhd8ed1ab_0  conda-forge
dagster-ssh                    1.5.5    pyhd8ed1ab_0  conda-forge
dagster-twilio                 1.5.5    pyhd8ed1ab_0  conda-forge
dagster-wandb                  1.5.5    pyhd8ed1ab_0  conda-forge
dagster-webserver              1.5.5    pyhd8ed1ab_0  conda-forge

While many but not all of the same dagster-* packages have the libs version 0.21.6 on PyPI.

dagster-airbyte: 0.21.6
dagster-dbt: 0.21.6
dagster-mysql: 0.21.6
dagster-papertrail: 0.21.6
dagster-spark: 0.21.6

dagster-airflow: 1.5.6
dagster-graphql: 1.5.6
dagster-pipes 1.5.6
dagster-webserver 1.5.6

Looking at the conda-forge/dagster-feedstock which produces these packages, it seems like the intention is for the immature library packages to have the immature library version lib_version (currently 0.21.6), but for some reason that's not working:

  - name: dagster-postgres
    build:
      noarch: python
      version: {{ lib_version }}
      script: cd dagster-postgres && {{ PYTHON }} -m pip install . -vv --no-deps --no-build-isolation

    requirements:
      host:
        - pip
        - python >=3.8
      run:
        - {{ pin_subpackage("dagster", max_pin="x.x.x") }}
        - psycopg2-binary
        - python >=3.8

    test:
      imports:
        - dagster_postgres
      commands:
        - pip check
      requires:
        - python >=3.8
        - pip

    about:
      home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-postgres
      license: Apache-2.0
      license_family: APACHE
      license_file: dagster-postgres/LICENSE
      summary: A Dagster integration for postgres

It also seems like the intention in the documentation about library versioning

Looking at the conda build docs on multiple outputs it does seem like the versions key is supposed to allow each of the subpackages to have its own version, but for some reason that's clearly not happening.

[sryza]

Hey @zaneselvans - I agree that it should show up as 0.21.x. Would you be able to report this in the conda-forge repo? Our build tooling only handles publishing directly to PyPI.

[zaneselvans]

Sure, I've just reported it over there.

[sryza]

🙏