Code Structure - Organization into a "Many Repo" (Breaking up the monorepo)

[DanVanAtta]

Let's experiment with multi-repo

I can see now a lot of benefit to going to multi-repo. I am feeling a bit burned by the lobby 2.6 rollout, issues with 'maps+lobby' server running out of memory and that being hard to diagnose, and finally not really getting the benefit of having everything on latest while still also having to maintain versioned dependencies.

I think mutli-repo can help us a bit here and get 2.6 released a bit faster and put us on a healthier long term path in terms of code organization and even contribution. I am planning to experiment with mutli-repo in relatively easily reversible ways and I'm happy to create additional repositories so that others can do so as well. I know there is already some support for multi-repo & a service like architecture, we could potentially move quickly. Please consider and provide feedback.

Going mutli-repo, I think each repo should have these characteristics:

• self contained as possible
• merges deploy a latest to production (eg: build latest installers, publish jar files to Github's maven dependency management system, deploy lobby to prod)
• any 'server' repo will have a corresponding 'client' repo. The 'client' repo is the one shared dependency between server code, and any other code that uses that server

For roll-out, I'm thinking:

• we'll keep "triplea-game/triple" mostly intact
• disable server auto-deployments from 'triplea-game/triplea')
• create new repos, copy code in to it, clean it up, get it working and deploying to prod
• first focus on the different servers that are integrated with the game-client via HTTP
• see how things go, next step would be to update 'triplea-game/triplea' to work with the new repo and delete any obsolete code
• then start more invasive break up of the 'game-client'

The 'bots' are not listed because I think next steps will also include for us to delete all of the headless code in favor of 'network-relay

Concern, roll-out delays and version 2.6:

I think we use this method to get out lobby v2.6

There is an issue between 2.6 clients and 2.5 & 2.6 bots. We can work on this in the meantime, there theoretically should be no issue with having a 2.6 lobby, 2.5 bots & 2.5 client.

Next, having a 'minimum' version notification built-in allows us to account for backward compatibility issues and have a 'turn-off' switch for old client versions. With this, the process to break off client functionality to a new server could become routine (see 'roll-out' above)

Concern, loss of code history

We'll certainly lose some code history. The main game-code will hopefully mostly stay together and be where we have most of the interesting history. Losing code change history in lobby and maps-server code does not feel too horrible.

Repo Dependency Diagram / Roadmap for consideration & feedback

triplea-repos.graphml.zip

[DanVanAtta]

cc: @RoiEXLab and @bacrossland , y'all both expressed support for a service architecutre and multi-repo, WDYT?

[DanVanAtta]

Got a start on this, some early examles:

• https://github.com/triplea-game/docs
• https://github.com/triplea-game/systems-install
• https://github.com/triplea-game/license

I'm liking that the '.github/CONTRIBUTE.md' file can be tailored for how to contribute to that immediate repository. Similarly the README can be very focused as well.

[DanVanAtta]

Discovered a significant draw-back to publishing 'jar files' to github packages - access requires an access token. It is not the end of the world, but it is a another one-time setup step before the project can be built. This page has details: https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-gradle-registry

[RoiEXLab]

@DanVanAtta I might be missing something, but can't you use the default secrets.GITHUB_TOKEN that is available by default AFAIK? We use it here for example:

triplea/.github/workflows/deploy-github-release.yml

Line 8 in 73b2fd2

GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Regarding your plans of splitting the repository: I'm not quite sure how to interpret your figure. What do the colors mean? What do the arrows mean? Are they referring to git submodules?
Regardless, while I do think splitting logically independent blocks to a separate repo is probably a good thing, I don't think it's necessary or maybe even desirable to go all out to break up the game into separate repos. If you break up the repos into too tiny chunks you'll end up having to synchronise multiple pull request in order to not break anything. I think the gradle config already provides us with a lot of benefits for a clear separation of concern (of course it's not perfect but we can easily fine tune it)

[bacrossland]

I like what you have proposed @DanVanAtta but the diagram is a little confusing. Is each color a separate repo or is each box a repo?

I 100% agree that splitting to different repos based on their role will help to rollout improvements to the overall systems in smaller increments. For instance, the recent forum bug holding up release of the game engine wouldn't happen. The issue would only effect the forum client and could be worked in isolation while the game engine updates could still rollout.

[DanVanAtta]

Re: access token for downloading maven package from github's maven repository

The publish of jar files can be done using the built-in secret token. I worked on an example to start seeing what this would look, here is an example: https://github.com/triplea-game/maps-client/blob/master/.github/workflows/publish-jar.yml

The above is running a gradle build (jar task) and then is uploading that to the packages list: https://github.com/orgs/triplea-game/packages?repo_name=maps-client

Eventually with the capability to use it like this:

<dependency>
  <groupId>org.triplea</groupId>
  <artifactId>maps-client</artifactId>
  <version>0.0.0</version>
</dependency> 

Notice that the above is a build dependency, and of course is not a project dependency. We need to add some small config into our list of maven repositories in our gradle build config. Access to the github maven repository requires an auth-token. So the context is from a developer machine running a build and pulling down dependencies.

Discussion Points / Q&A

What do the arrows mean? Are they referring to git submodules?

Each card/item in the diagram is a repository. Each repository will have an output artifact. Either it will be a game-installer, a maven package,or a service that is deployed to prod. The lines represents a "uses" relationship between repositories, the direction of the arrow points to the item being used.

For example, the 'lobby-server' will have a maven Jar file dependency on 'lobby-client', and the 'game' will also have a Jar file dependency on the 'lobby-client'.

What do the colors mean?

Green: The 'green' repositories are the front-end, generally what we would consider to be 'triplea-game/triplea', the code that will be in those repo's likely would be what we had in 'game-core' and 'game-headed'

Yellow: These are repositories that are "stand-alone", typically they host non-java code related stuff.

Purple: These are all servers, generally would be separate drop-wizard servers each with their own JSON-over-HTTP API

Blue: These are the remaining items and are the various http clients for each server. Part of the idea with the http servers is that they will be developed in parallel with a stand-alone 'client' jar that presents a full java API to the server. So, all of the wire-mock & feign code will wind up in these 'blue' packages, and would present a pure java API to anything that would want to access a server.

Game split into different repo's

I don't think it's necessary or maybe even desirable to go all out to break up the game into separate repos

I generally agree. Though, we may have different perceptions of what "all-out" means. My initial target is to split the game-engine into 3 repositories. This is with the goal to force ourselves to partition the code. If we split the AI further into their own repositories for each AI, I wouldn't quite consider that 'all-out' just yet (notably because we ideally have zero dependencies between AIs and they 'should' be completely standalone dependencies). With that said, something like a 'AI-lib' is probably not desirable and indeed we do not want to go crazy with a dependency tree that would require 5 PRs to update an AI. The repositories would be there for places where we really want to enforce a very hard line between dependencies .

The game-engine has the luxury of a lot of time before we're ready to do much in the way of significant splits. I'm thinking we will continue to use sub-modules quite heavily. There is a still majority of code in the game-engine and that will likely continue to be the case for a while. We have a lot of time to design further segregation of the game-engine code. The goals of any first moves would be to "force" break any code dependencies.

[DanVanAtta]

Q: Is each color a separate repo or is each box a repo?

Each box is a repo. The colors are for grouping.

Benefits of Split

For instance, the recent forum bug holding up release of the game engine wouldn't happen.

Indeed! Further, our desire to move forum in docker would be facilitated even. I just learned that Github 'packages' can host docker images. Hence, our 'forums' repository could package a docker container and then trigger an update/deployment whereby the forum server pulls a docker image from github.

I think a big gain would be for AIs as well. Having those in their own repository forces us not to use AI code in the game-engine, and we can create more granular access permissions, and the partitioning just more re-inforces us to respect a larger system design.

[RoiEXLab]

@DanVanAtta Thanks for clarifying. I like your proposal in general. We'll have to discuss the minor details, i.e. where to split the repo once we got to this point. But for now if we start with just some separate repositories we can easily re-evaluate on a case by case basis where more segmentation is useful and where the separation is just right.

[DanVanAtta]

Adding github access token in order to run gradle build

Here is a PR adding the dev-setup steps required to download maven packages hosted on github packages: https://github.com/triplea-game/docs/pull/2

After going down this path, every developer would have to follow that in order to run the gradle build

[DanVanAtta]

Re-considering multi-repo

In short, I think perhaps our best bet instead of doing mult-repo is
to do a better job of modularizing the mon-repo. Multi-repo is not a panacea.

Notably, I just learned that there is a 'paths' configuration option
for github actions. That is a bit of a game-changer. Using that we
can use module specific builds & deployments, which were perhaps the key
benefits of going multi-repo.

Multi-Repo Status

I made it somewhat far down the multi-repo path. The 'maps-client' repository
is created, it posts a JAR file to the github maven repository on each build,
and there is a 'maps-server' repository that pulls this in.

Mono vs Multi Repo - considerations after having done some experimentation now

With more experience with how it'll look to have multi-repo for TripleA, there
are some deeper understandings about the pro-cons.

To start with the pros.

Multi-Repo Pros

• isolated & smaller code bases
• faster and more targeted builds
• independent deployment that makes sense (no deploying lobby just because
  documentation has changed)
• more targeted README documentation for each repo

I think the noteworthy thing here is that having conditional github actions
based on path delivers most of these benefits.

Multi-Repo: deeper insights (the cons)

Access Token

The biggest drawback is the access token needed to download dependencies.
It's already very difficult for new developers to get started, having a
401 downloading dependencies out-of-the-box is going to be just another
problem. Further, the configuration needed in gradle and overall to support
this is not just a one liner. It's not a lot, but it's still somewhat significant.

Lots of Tooling De-Duplication

Things like 'code-cov' and any other static analysis tool needs to be configured
on each repo. Config files that drive those tools (like pmd.xml) is duplicated
mercilessly across each repo. The gradle config to enable stack trace output in
tests, build shadow jars properly (which was super painful to figure out) is
all quite significant in sum.

Distributed Pull Requests

This looks like it could become quite a problem. There is no single view AFAIK
to view all pull requests within an organization. The idea of making sure that no
repo has any open pull requests is somewhat daunting.

Code isolation is too extreme

This is a pro, but multi-repo starts to go into an extreme. Either we start
having shared repo's for things like web-server common tools, or we duplicate
those. The duplication in one way is a good thing, but we are at an extreme
end of not allowing for sharing of tools between modules and that is a drawback.

Dev Setup Instructions & Docs get weird

It's weird to have a repo dedicated for docs & to have each README duplicate
the same "look to this repository for the full dev-setup instruction". Not
being able to update the central docs repository with pull requests that also
update architecture and features at the same time seems like a drawback. Mostly
though having the docs be divorced from everything else in an isolated repo feels
like more a way for the docs to become stale and useless than anything else.

Overall effort to effectuate this change

It has not been all that quick to drop existing code into new repo's and have
everything else work. There was a good bit of setup with having template repositories,
tooling configuration, a brand new gradle config, and then getting the drop in code to work.

Lack of infrastructure shared tooling & duplication of server lists

I set up a 'systems-install' repo which is meant to be ansible deployment that will
do a systems baseline setup. It has its own inventory file, which is fine, but the intent was
to have each repo also be "self-deployable". This means that we would have each repo just
deploy the application part of what is in that repository. Though, we can't easily re-use
infrastructure roles in a multi-repo, and having an inventory list in the server-setup repo
and the application repository is an anti-pattern. Inventory lists are really best when
there is just one of them with unique listings of the hosts. Having to first add a server
to the 'setup', run that, then add the same server to multiple repository inventory lists
is not great. Making updates to this is also not going to be great.

Not a convincing model for new contributors to be impactful

Particularly for new contributors, I fear the amount of frustration that would be experienced
by trying to do incremental pull requests across N repositories with merge ordering dependencies
& updating dependency version numbers along the way. Grant-it, I hope that "heavy" contributors
will gain write access much faster going forward, but it's still seems a major concern
and I have trouble thinking anyone will even dip their toe in the water of this because it's
seemingly going to be so obviously painful.

Summary - Let's go for Better modularization within a mono-repo?

In sum:

• reduce build times
• modularize builds & deployments within the the mono-repo
• make it easier to get started and just contribute to the game engine
  without being forced to branch out into all aspects of the project

Using the 'path' attribute on github actions I think is a game-changer and
can give us most of all of the benefits of multi-repo while staying in mono-repo.

I think if we try to modularize code better/more, that will be the most important.

Sticking to a mono-repo gives a lot of benefits, one place to view pull requests,
one place for documentation, no version management between modules, one pull request
to make cross-cutting changes.

For stronger modularization, I think the idea is going to be to try and avoid shared
libs. For example: instead of sharing 'java-utils' project between the game client
and server, we would only use it within the game client. Another way to view this,
the github action files will specify which paths are triggered, generally we will
want an application to be as self contained as possible within its folder (so we
shouldn't really have a github action that lists 5 different paths, but instead
have one path for the game-app that has very few dependencies outside of the game-app
folder). With that said, I do think we still want the general model that the game
will depend on http-clients, and servers will depend on http-clients, and that will
be the shared library overlap. To emphasize, what that does exclude are the larger
shared libraries for generic tooling, and instead we'll try to make those tools
to be game-client specific and contained with the game client, or contained within
the servers.

For a deployment & infrastructure level, I think we should try to modularize
the deployment to be within modules. A common systems setup folder seems inevitable,
but it's easier within a mono-repo to fully specify the server setup in a modular way.
For example, the node-bb deployment could be almost fully contained within a '/node-bb/deploy'
folder, with exception that the inventory file would be in a '/servers/inventory/prod' file.

Next Steps

This is a longer read, so I appreciate anyone sticking with it and the full consideration.
I think next steps are going to be to immediately try and improve the modularity of the
current mono-repo. I think we really need to get the build times down, we need to improve
build reliability and simplicity. Most new contributors want to dip their toes in the water
and just update the game. Having new contributors install docker, be forced to know the full
architecture with server interactions, run a 5 minutes long verify script seems like a very
bad state. I don't really know how to lower the barrier-to-entry for new developers, but it's
something that feels extremely important.

I'll leave the additional multi-repo's up and available for a little bit, but will plan
to clean them up after a few weeks.

[RoiEXLab]

@DanVanAtta Didn't know about the paths attribute, seems like a really good idea using that 👍

[DanVanAtta]

I don't know how new the 'paths' attribute is, but I only learned about perhaps 2 days ago!

This article: https://buttondown.email/blog/just-use-a-monorepo was linked from hacker news

I'm not sure if the article is necessarily all that compelling, but it was interesting to think that we were going the opposite direction. In the comments of the article, someone gave an example of the 'paths' attribute where I then learned about it: https://news.ycombinator.com/item?id=34359736

That paths capability really does seem like a complete game changer for this whole conversation!

[bacrossland]

The path attribute in github actions is nice. It makes CICD based on changes to certain folders data easier. Using it to format and deploy updated documentation is a common use case. No need for full deploys when all you need to do is fix docs.

Separate repos: I get that changing the gradle and build pipeline can be difficult. For a lot of large projects that have the same dependencies this one does (and the same issues) they got around that difficulty by using git submodules. Each module was in a separate repo that git submodule knew about. Devs only needed to check out the parent repo and then run the submodule checkout. It would git checkout the source for the other modules and they would be there in the local parent repo just as if they were stored in the repo. You make changes to the submodule, check it in, and later update the main repo to point at the commit hash of the updated submodule. The last major project I worked on managed 50+ submodules this way. If you are curious to know more, here is a 15 minute primer: https://www.youtube.com/watch?v=gSlXo2iLBro&t=71s

Docker and lowering the barrier of entry:

Having new contributors install docker, be forced to know the full
architecture with server interactions, run a 5 minutes long verify script seems like a very
bad state.

When it comes to lower the barrier to entry on a project, Docker is fantastic. With docker and docker-compose you are not forced to know the full architecture because it's already handled for you. Instead of telling a dev that they need to download, setup, and configure a database on their local system, they use one command to pull an image and launch it as a running container. Need to upgrade? One command to pull a new image. Need to wipe the whole thing? One command to remove the container. It makes working with dependent systems easier because you don't have to know and manage that dependent system.

Take NodeBB for instance. Unless you use that library or NodeJS as your daily tech stack, why would you want to install and maintain a NodeJS install, a NodeBB install, and a Mongo DB install on your dev system just so you can fix an issue between your Java application and the basic version of NodeBB? A docker-compose file that spins up NodeBB and Mongo DB using the correct NodeJS version is a single command to up and down the stack. No more worrying about needing to know Javascript and NodeJS. Just get on with fixing your Java application.

A 5 minute verify script sounds like a perfect candidate for refactor. If the script is a necessary step in the build or test pipeline, then it should be looked at to see if it can be faster. Github actions can run that script on a developer's branch on them checking in the code before creating the PR. That will save them from needing to run that locally as part of their development process.

I've just changed jobs which should free up some dev time for me. I can help with lowering the barrier which will make finding more devs easier and relieving some extra load from you @DanVanAtta and @RoiEXLab.

[DanVanAtta]

I compiled a project list very quickly here: https://forums.triplea-game.org/topic/3388/triplea-dev-project-list-jan-2023
I certainly had a small moment of despair & panic thinking about how long it would take to accomplish those goals and how hard it is has been for others to make substantial progress. The help is invaluable @bacrossland.

My previous post was a bit long and super detailed. It's hard to convey everything and be concise. I think there are three key problems with multi-repo (perhaps lost in the details):

• (this is the big one) to share JAR files via github, it requires a personal access token to be configured by every developer for a build to succeed. It's a real shame that github does not provide simple HTTPS access to its maven repo where Jar files can be stored, but requires that access to be authenticated
• tools have to be configured N time, thinking of things like PMD, codecov, codacy (anything that needs a config and/or integration with the github repo).
• managing pull requests, and perhaps the general drawbacks of submitting multiple PRs, incrementing versions (and doing that while waiting for CR)

Long story short, seemingly the path based build gives us so many of the benefits of many-repo.

For reference, a sample build script for a 'client' library (something that has no other project dependencies) can be seen here: https://github.com/triplea-game/maps-client/blob/master/build.gradle, it's pretty lengthy and all boiler plate (minus the dependency list, which is not too horrible, but that's a fraction of the overall build script; also take note of the customizatoin around the shadow jar task so that the jar file is both packaged properly & also well named). An example of a 'dependent' library (maps-server) can be seen here: https://github.com/triplea-game/maps-server/blob/master/build.gradle, there it can be noted each dependent project has to be listed as its own repo and there is config for configuring access tokens.

Re: docker & barrier to entry

I totally agree on the reduction of the barrier to entry provided by docker. To try and clarify, I've found a lot of contributors simply and only want to contribute to the game-app part of TripleA. They just have no interest (yet) to work on the lobby part. The issue is that it's important to be able to run a full CI verification locally, and that means everyone had to run the full set of lobby tests & setup whether they were updating the lobby or not. That is where a lot of the time was sunk during builds.

I've been working on this today, the new verify script for the game-client will be at: ./game-app/run/check, so far it's taking 28s (on a AMD Ryzen 9 7950X with 64GB RAM) and probably can be cut in half. The lobby tests could not be run in parallel as well, but we can run the game-app tests in paralell. We get this same benefit of course with many-repo.

The docker DB will certainly still be a factor when doing server-side work. Totally agree that is light-years nicer than each developer having to install a DB to bare-metal, let alone considering the effort to destroy it and rebuild it.

Re: A 5 minute verify script sounds like a perfect candidate for refactor.

Indeed. Getting the build times faster is generally always high value work. It helps everyone, many times. We have a number of tests that can be simply speed up. The lobby side testing was a large chunk of that 5 minutes, and the tests were forced to run in sequence... I agree it's a good place to spend some effort to improve.

[DanVanAtta]

One negative thing about path specific builds so far seems that the coverage report in PRs could be very wonky.

Currently coverage is now only computed for master branch builds so that we can get an overall trend. Slower master branch builds are not that important, I'm on the fence if it is worth it.

One nice thing about multi-repo was that we could enforce code coverage requirements readily. Perhaps there is a way to configure that in a path specific manner.

[DanVanAtta]

Good news, just landed a PR just now that did not update any source code, and no tests were run! We are in a place now where the CI does not run if for example only the docs are updated.

#11379