Polylith monorepo

[TimoKramer]

We want to switch to a monorepo with polylith. That enables us to improve workflows and testing because we have all in one repo.

[TimoKramer]

Discussion in polylith on the problem of releasing libs from a polylith-repo: polyfy/polylith#212

[TimoKramer]

I was going through some migrations to polylith and there are some issues:

• every artifact needs to be released in lockstep
• dependencies are integrated implicitly
• we have to use a top-namespace and change all the namespaces (breaking changes)
• we have to introduce interface-namespaces which introduce breaking changes as well

[TimoKramer]

polylith offers a much better way to keep all datahike-artifacts in check but it will bring breaking changes and maybe unforeseen problems. so far no one afaik used polylith for OSS libraries.

[whilo]

Thanks for investigating! What alternatives exist to achieve similar integration? Can we for example have downstream integration projects that run their tests automatically? There are benefits with not putting everything into one repository, replikativ projects stand on their own compared to an "unbundling" approach where you take a large project and allow people to use parts of it as libraries.

[jsmassa]

There is always the option to build something ourselves as we need it. Running integration tests is the least of our problems since they can simply be triggered by a GitHub Action in the pipeline and disabled easily when we decide to solve a problem later.

As I remember the most important reason that drove us to consider polylith in the first place was a repeating issue with inconsistent dependencies and a bad developer experience when we needed to change something in hitchhiker-tree or konserve to solve a problem popping up in datahike.

Another consideration was that with polylith, a project that wouldn't need all of datahike, e.g. a remote transactor, wouldn't have to pull in the whole thing but it would be possible to only get the components that it actually needed - which is the whole point for using polylith.

I think we can prevent the issues we had in the past by improving the ci pipeline, doing regular updates, and having an eye on the functions to be used from outside, i.e. the functions in the api namespace in datahike, so we don't introduce breaking changes anymore.

It is still more work though to maintain many projects instead of just one, so it still makes sense to consider combining projects with polylith.

[TimoKramer]

We are working with a monorepo now