Moving towards monorepos?

[mickael-menu]

Currently we have:

• one repo per library (streamer, navigator, shared, lcp and opds)
• one repo of a test app using these libraries
• (at least for mobile) a "workspace" repo embedding all the other repos as submodules

I don't think that this approach is really beneficial for R2, and it has many downsides:

• Managing many repos is more cumbersome.
  • Releasing a new version of the toolkit means tagging and testing all these repos at compatible commits.
  • Issues sometime overlap several repos, and it's less easy to search through them for users.
• It's quite confusing for newcomers, see for example this recent Slack message:

I discovered Readium today and I think it's exactly what I need in my app. [...] I read the docs and the project is still a bit difficult for me to understand with all those repo github and standard.

• It's a bit confusing for contributors too. For example, where to put the documentation? We could have specifics in each repo, but what about overview and user guides? Putting them in test-app doesn't feel right, and splitting the documentation everywhere makes it less discoverable.
• We can't easily share resources between projects, such as test fixtures.
• All the individual repos are disconnected, which means that we can't use git bisect to track down regressions.
  • If you're not familiar with git bisect, basically it allows you to find which particular commit introduced a bug in a few minutes. Without it, we work blindfolded and pinpointing the source of a bug can take hours.
  • Theoretically r2-workspace could be used for that, but in practice we are not making a commit in r2-workspace for each commit in all the other repos. It would be a nightmare to merge.
• While we can use the libraries independently, when you want to pair them, you need to use compatible commits/versions. Currently, besides using manually tagged versions, there's no way to know if 2 commits of different projects are compatible.

Any non-trivial change made in the toolkit needs joined PRs in several projects, as you can see in this screenshot:

Using a monorepo per platform could fix all these issues. And this doesn't mean R2 libraries can't be used independently, since we can have several modules in the monorepo, for example:

• ExoPlayer in Kotlin has a core module, and then optional extension modules.
• Retrofit in Kotlin has a core library, and then optional converters and adapters libraries.
• RxSwift in Swift contains several libraries: RxSwift, RxCocoa, RxBlocking, etc.

Some downsides of migrating to a monorepo would be losing the Git history and needing to migrate all the GitHub issues. There are some solutions to keep the Git history though, which might be worth trying out.

If you're open to the idea, I can put together a proof of concept to see how it would work.

[aferditamuriqi]

when we originally started all the modules were in one repository, and we had to split them into several repositories

[mickael-menu]

when we originally started all the modules were in one repository, and we had to split them into several repositories

Why did you have to split it?

[HadrienGardeur]

The philosophy behind R2 has always been that you can pick and chose what you want to use instead of having to use a monolithic SDK.

This means that some projects might just use the streamer, while others would use the streamer plus multiple navigators for instance. We've seen this with native apps based on Web frameworks for instance.

In Go we never separated shared from the streamer because it's very easy to use Go packages to strictly use part of a repo.
If we can have something as easy to use in Kotlin and Swift, I'd be opened to this idea, but otherwise I'm wary of this approach.

[mickael-menu]

In Go we never separated shared from the streamer because it's very easy to use Go packages to strictly use part of a repo.

Yes, I believe we can do the same thing in Kotlin/Swift, as demonstrated in RxSwift and Retrofit which I mentioned above.

I understand the need to keep each library independent, and I'm not questioning this. The idea is to have the same thing, but in one repo. Each library would be split in different folders/modules and have their own Gradle/Podspec/Package.swift file to be distributed as independent libraries.

To be more precise, I see things like this:

• One r2-swift/kotlin repo containing the libraries shared, streamer, navigator, opds and lcp.
• The r2-testapp repo which would replace r2-workspace and have the monorepo as a submodule. This would reflect better an integration from an actual reading app than having a r2-workspace.

Separating the codebase in several repos might have encouraged contributors to keep things loosely-coupled, but I think that this should be enforced with a clear–cut architecture/interfaces, and code reviews.

[aferditamuriqi]

I think one of the big differences between our R2 modules and the examples used (RxSwift and Retrofit) to compare, is, that R2 modules are also used as submodules in implementations.

we have had this discussion in the engineering calls more than once about the testapp and workspace etc.

I would not be in favor of moving all into one or changing this structure. there is a lot changing in the modules itself, maybe revisit this at a later point. refactoring the structure as it is right now would for sure cause issues in current implementations and should be thought through if even considered.

[llemeurfr]

From another angle, in order to ease discovery for newcomers, we have discussed the creation of a documentation repo for Readium Mobile iOS, Readium Mobile Android and Readium Desktop (note that good Readium.org project pages could achieve the same goal).

[mickael-menu]

I would be fine revisiting this later, maybe once the public APIs are stable.

However, I want to stress that these issues are not small. Personally, I find that not being able to use git bisect is particularly handicapping. I didn't see any convincing counter-arguments since a single repo is fully compatible with independent libraries.

[llemeurfr]

Well, lots of work to achieve that monorepo move vs lots of work planned on the toolkits this year for getting all the features we want is a counter-argument, sort of (with my project direction hat).

[mickael-menu]

Well, lots of work to achieve that monorepo move vs lots of work planned on the toolkits this year for getting all the features we want is a counter-argument, sort of (with my project direction hat).

I think this wouldn't take more than a day though.

But yeah, I didn't expect that we would do this any time soon. I just wanted to test the waters and see if I missed something in my reasoning.

On the long term, I think the time spent migrating to a monorepo would be repaid in full by finding regressions much more quickly.

[jccr]

I'm very familiar with JS monorepos.
IMO difficult to setup properly, but worth it in the end.

(Not throwing any opinion on whether we should or not for Readium)

[danielweck]

As "casually" discussed in previous conference calls (i.e. no firm consensus, just exchanging ideas): a monorepo for the r2-xxx-js GitHub repositories makes sense indeed, as each R2 "component" would in fact remain a standalone NPM package, each with its own semantic version (this would be an opportunity to migrate the existing packages under the @readium/ "namespace" e.g. scoped packages @readium/shared-js, @readium/navigator-electron).

Needless to say, the interspersed Git commit history would need to adhere to a strict PR + merge protocol, for example Git commit messages with non-ambiguous namespace scoping, using conventional-changelog / standard-version syntactical conventions, etc. The Git release tags would need to convey state about multiple packages under the same codebase "umbrella". Plus, automated CI builds would need to be (re)scripted carefully to generate individual build artefacts under a common roof, to deploy to test servers (i.e. streamer), etc.

Yarn workspaces are pretty much essential, although there are alternative options. Lerna can help too.

This would be quite a disruptive endeavour in our current roadmap / work flow, due to all the plumbing that would need to be implemented and documented. Probably better engage in this process when / if we refactor the TypeScript / Javascript codebase to better implement "isomorphism" / multiple runtime targets (e.g. NodeJS, Electron, Web).

[mickael-menu]

Thanks Daniel, I didn't think about the impact on Git tagging/commit messages. If we go down this road, we'll definitely need some structure. In my experience with the mobile toolkit, often commits in different repos are actually inter-dependent – as shown in the screenshot I posted – which is the main reason why I think this should all live in a single repo. But sometimes, there are changes concerning only a single lib, and it would be worth marking them as such.

The roadmap is pretty packed for all platforms this year, but I'll be happy to revisit this later when I'll have more bandwidth to see how it would look like on the mobile toolkits.

[jccr]

I have a first draft of how this could be for the web project:
readium/web#1

[qnga]

I just finished a big refactoring and I can confirm that working with multiple inter-dependent repos is painful and time-wasting.

[mickael-menu]

@qnga Thanks for sharing your experience with this.

I found another grievance with the current situation in Swift: the Travis CI build is basically useless as soon as we depend on changes in r2-shared-swift. Because the Travis build uses the Carthage dependency for r2-shared-swift, it fails when it's not the right version. We could use develop instead of a fixed version in the Cartfile, but this wouldn't solve the problem in feature branches. So it's constantly red in my PRs.

[mickael-menu]

Another feedback shared by a contributor on Slack:

Yeah I think that would make sense, whenever there is an issue I am not sure where the problem really lies and need to search in each repo

[stevenzeck]

I agree in that the R2 frameworks should be kept modular, so integrators only need integrate what is needed. However, as @mickael-menu pointed out that can still be accomplished by having a single repository instead of several. Whether or not they are split into separate GitHub repos only affects a couple of groups: Contributors, and integrators that need to report an issue.

As a contributor, it's painful to have to switch branches of every module to test out something when it stretches across multiple. Example, The Search Service code for Kotlin had changes in all of the modules and it was a pain having to go back and forth between branches trying to do something. Having a single repository would eliminate that pain point. Will the git history be more complicated? It's possible, but features will be centralized to just those commits instead of commits in several repos. Will issue tracking be more cumbersome? Again, it's possible, but right now a lot of issues are just opened in the testapp anyways because either the reporter doesn't know which module it belongs in, or it belongs in multiple.

[mickael-menu]

Will the git history be more complicated? It's possible, but features will be centralized to just those commits instead of commits in several repos.

Right now we have commits that are pretty disconnected, for example this one in LCP, required by a change for the search service PR: readium/r2-lcp-kotlin@f622669

So it would probably improve things. We could also use prefix tags such as "opds: Improve parser" in commit message to help skim through the git logs.

Will issue tracking be more cumbersome? Again, it's possible, but right now a lot of issues are just opened in the testapp anyways because either the reporter doesn't know which module it belongs in, or it belongs in multiple.

I also think it would improve issue tracking actually. We need to use ZenHub to have access to a unified backlog for all related repositories. This is not visible for contributors unless they also install ZenHub. With a single repo we could just use GitHub project and keep everything publicly visible.