Open points to discussion

[tomsej]

Hi @DavidVujic, I have been playing with python-polylith, and I am truly impressed. It is really close to how I think that ideal Python project would look like (at least for me). I have two points I would like to discuss:

1. Decoupling of base, components and external packages require some testing of the resulting artefact (whl or sdist). Because it is super simple to forget to include some brick or external package and find out that app is not working in deploy or even in prod. Since I am using Docker for all packages, I am trying to mitigate this risk in the Dockerfile:

ENV TEST_CODE="import sys\ntry: import namespace.package \nexcept Exception as e: sys.exit(e)"

COPY ./dist/$WHEEL /tmp/
RUN pip install --no-cache-dir --upgrade /tmp/$WHEEL --target ${TARGET} && \
    echo -e $TEST_CODE | python

But it is less than ideal. This is not a case for the compiled languages, I guess. Think the best way would be to running tests in the fresh venv where wheel artefact is installed. But again, the problem is with relative paths and specification of test that should be executed. Do you think this should be somehow supported by multiproject plugin? Or it is more up to CI/CD to manage this? I am more keen to CI/CD path, but it would be good to warn users that this might be tricky (especially with bigger projects with multiple dependencies).

2. In monorepos you usually have "universe of dependencies". If I get it, in polylith every project is totally independent to each other. So in one project you can use e.g. pandas==1.5.0 but in the other pandas==1.1.0. But it can also mean that same component in both the projects can behave differently (since it is using different external dependency). Any idea how to tackle this?

Thanks so much for this!!!

[DavidVujic]

Hi @tomsej! Thank you for your feedback, it is very appreciated! ⭐

I am planning to add some help for the thing with making sure a project has all the bricks in place - something like a poly check command. I'll have to refresh my knowledge about the original Clojure poly tool and how it is done there 😄

The second part - I also plan to add tooling support there for components to specify their third-party dependencies. Something like a deps.toml file that the tooling (either Multiproject or Polylith) can parse external dependencies from. I think that could solve many versioning issues, if a component can decide itself what is needed, instead of trying to manage it in the project.

[DavidVujic]

Until the new tooling features are in place, I think testing the built artifact in Docker seems like the way to go.

Also, I think - but I have to double check that - it is possible to run tests for code in a specific project without building it into a wheel or sdist, even with relative paths.

I am very curious about your project, is it by any chance Open Source? It would be very interesting to see the setup! If you have more ideas or thoughts, just add them here. That kind of feedback to a new project like python-poetry is really valuable. Thank you again!

[tomsej]

Hi @DavidVujic, unfortunately I am unable to share the project since it is private one :/. But feel free to ask any question. We are currently trying to switch from Pantsbuild to your Polylith setup because Pantsbuild is just too heavy and "magical" for us. Polylith seems to work for us so far, so you can expect more ideas and feedback from us in the future.

Regarding the tests. We are not using Poetry in-project environments, since we have everything in devcontainers. So, we do not have any specific project venv, only the global one (the reason for that is performance, since bind volume is way slower in MacOS). That means that your proposed solution would not work for us.

It would be wonderful to have poly check command. Let me know if I can help.

Regarding the deps.yml I saw that in Polylith documentation. It would be great, but not sure if the new way to specify the requirements should be introduced. Couldn't we use pyproject.toml in the components and then in the project add this component by running poetry add ../../component/xxx? Unfortunately, this is not working now, since Poetry does not allow relative paths. Another idea I had was already mentioned universe of dependencies. It is a way that Pantsbuild is using. You would have all the dependencies specified in the root pyproject.toml (let's call that master dependencies). And poly check command would check if any project dependency differs to master one? And possibly fix it up?

[DavidVujic]

Great ideas, thank you! ⭐

I'll think about this and see how the issues can be solved. Currently, I have some half-started ideas/scratches in Python code. I should focus on one thing at a time and solve it step-by-step 😄

[DavidVujic]

Hello again! 🎅

I have added a new command in the Multiproject plugin, to check for missing dependencies or packages in a project. It uses MyPy behind the scenes. I plan to build a poly check feature on top of this one, to make it more User friendly and with a good fit for Polylith 😄

Meanwhile, I hope that the new poetry check-project command from the Multiproject plugin can be useful.

EDIT: 🤦‍♂️ I was a bit too fast with deploying a non-working new command. So, this one is not very useful yet.

EDIT 2: there's a new version of the poetry-multiproject-plugin that will perform checks on a project. It's not 100% accurate, but should catch missing library dependencies (unless you also have them globally installed in the machine), and missing package includes that are used in the code. More info here: https://github.com/DavidVujic/poetry-multiproject-plugin#what-does-it-do

[DavidVujic]

I've just released a first version of a poly check command. It relies on having the latest version of the Multiproject plugin installed. I would love to hear your feedback about this feature!

[DavidVujic]

I'll close this issue, because I believe it has been solved by the commands added. Please open a new issue if there is something I have missed or misunderstood!

I would love to hear your feedback about the poetry poly check and poetry poly libs commands here in this repo, or maybe in the pyslackers Slack? I hang out there and occasionally discuss Polylith and functional programming.

[tomsej]

I am really sorry. You wrote it during the Christmas and I forgot to write to you back. I do not actually use a poly check command, since it was kind of slow in our repo (maybe using a Ruff as a linter?) and produces some false positives (would be able to fix that with better mypy settings - we use pyright). So, we still run testing in Docker.

I am super excited about poly libs command!!!

[DavidVujic]

yes, the poly check command is kind of slow! Good idea, maybe Ruff could be used here. I'll try that one out.