Package has 0 dependencies

[dinjazelena]

Hey,

when u publish to pypi, only main dependencies are installed(dependencies under [tool.poetry.dependencies].
Currently when i pip install quinn, it does not install any library:

Right now u have all of your libraries under dev-dependencies, which is installable only by poetry for development purposes.
Anyway dev-dependencies will be depracated and u must migrate to poetry groups such as linting group, testing group, and docs group.

But anyway most important thing is that all libraries that quinn package depends on must be under main dependencies where now u have only python version specified.

I can open PR and fix all of this..

[SemyonSinchenko]

Quinn was designed for using on PySpark clusters where all the spark binaries and also PySaprk bindings are already installed . So, for me it is a right behavior cause I do not want to install another version of PySpark on my cluster.

For me it is not an issue. @MrPowers what do you think?

[MrPowers]

The only dependency that we would consider changing from a dev-dependency => a regular "dependency" would be pyspark. Dependencies like pyspark and chispa are intentionally marked as dev-dependencies.

I think @dinjazelena is right that technically speaking, pyspark should be considered a regular dependency (not a dev-dependency).

We want pip install quinn to work in a variety of execution contexts like Databricks notebooks, EMR notebooks, locally, etc. If I recall correctly, including pyspark as a dependency causes problems with pip install quinn is run in those execution contexts. Feel free to correct me if I'm recalling incorrectly.

[SemyonSinchenko]

We can make pyspark an external dependency, so for example we will have the ability to run pip install quinn[pyspark] and it will install pyspark. But we will still be able to just run pip install and simplify the dependency resolution process and also not break managed Spark environments like Databricks. It can be done via poetry groups I guess.

[dinjazelena]

Aha, so u expect this to be installed in place with already defined environments like DBR. Then yeah, as long as u test for all possible spark versions, good to go.

Still dev-dependencies will be deprecated soon:

Ideally you should have:

development group: pyspark or as Sem mentioned as extras so installable with pip install quinn[pyspark] for example
testing group: pytest, chipsa, pytest-describe
linting group: black, ruff, mypy
docs group: mkdocs thingies

whole dev project is installable then by:
poetry install --with=development,testing,linting,docs

Which helps u then in CI for example for linting u only need linting group or in docs u only need docs group, u get the point.

Would u like to change it in this way? With Makefile as set of common commands.

[SemyonSinchenko]

I like this idea!

[MrPowers]

@dinjazelena - sounds like a good idea, can you please submit a PR?