Unify shared dependencies

[mgorsk1]

Signed-off-by: at91mm mariusz.gorski@ing.com

[dorianj]

I'm not familiar enough with python packaging to have an answer here, but it is a bit of a shame we can't use constraint files -- it feels like the "correct" solution. But I do agree it's not intuitive, and even if we published build scripts everywhere, many people wouldn't use them because there is an expectation you can just pip install -r requirements and be off to the races

If instead of including requirements.txt, we only used setup.py deps + constraint file, i wonder if it's possible to automatically bring in the constraint file in setup.py or setup.cfg ?

[mgorsk1]

as far as I know (and from research I did) there is no way to make setup.py constraints aware. what you would do in such case is just add shared dependencies to install_requires without pinning versions and then, when doing pip install you supply pinned versions using --constraint parameter. what's important is that --constraint doesn't force installation of additional deps - if something is listed in constraints file but not in install_requires then it won't be installed

[mgorsk1]

to be completely precise - I also was thinking about making clearing setup.py files out of deps and pull them by reading requirements*txt files. again, it would be super easy for anything apart databuilder

[Fokko]

I think this is a great idea. I have one suggestion by splitting out the test-dependencies using pre-commit. LMYWYT

[sewardgw]

I think the idea of shifting the build context up one directory makes sense, it also gives you the ability to copy in the amundsen common that is on the same brach (not in pypi) and copy it into each image. This could also give the benefit that we wouldn't need to add new versions of amundsen-common to pypi.

[mgorsk1]

How would we go about pypi installation of amundsen services if common is not a pypi pckage? I think not all users use pre-built images.

Moreover, we have introduced apache atlas utils recently as a part of common and will be relying on them for both databuilder and metadata.

[sewardgw]

Is the pypi installation of search, metadata and frontend really that common of use-case? I get that we wouldn't want to pull the plug on that user group, and by no means am I against keeping it in pypi and just overriding it during the build process, but it just feels like one of those areas where we shouldn't need to have the common directory in pypi any more since it's now in the same branch - mostly thinking out loud here.

[verdan]

@mgorsk1 how would that work with the PyPi versions of Amundsen packages?
PyPi requires the files needed for the setup.py in the build, that means we need to include the requirements-common.txt and other common requirements file within the build for each package.

[mgorsk1]

That's exactly how it would work. The requirements-dev and requirements-common files would be a part of artifacts built by sdist and bdist. It should be possible with proper manifest definitions (assuming your concern is over symlinks resolution?)

[feng-tao]

how about databuilder? currently the model is defined in both common and databuilder which we should eventually make databuilder depends on common model to make it consistent to avoid model drifting.

[mgorsk1]

In principle I agree that long term there should not even be models folder in any of the project except for the common. This would be however a major refactor from a different angle than I had in mind for this RFC. I will add this to future possibilities.

[verdan]

this looks good to me and really excited about this 🙂
I don't know the technical challenges regarding the PyPi release yet and adding links to files outside the root directory, but more than happy to at least give this a try.
The next step would be to remove all the local models from each package and simply rely completely on common models.

[mgorsk1]

Nice, thanks! I've already started to draft something under amundsen-io/amundsen#1163 , seems most major things are resolved - only testing of pypi artifacts and building docker images remains.