Discussion on improving Cljdoc infrastructure & operations

[rvalentini]

Here we can gather ideas on how to improve the current infrastructure setup and simplify operations.

Discussion results should be documented in form of ADRs and actionable issues.

Status Quo documentation: https://github.com/cljdoc/cljdoc/tree/master/ops

[rvalentini]

Topic: Simplify the Cljdoc deployment setup

Documentation: https://github.com/cljdoc/cljdoc/blob/master/doc/adr/0017-use-nomad-for-deployment.md

[martinklepsch]

Google Cloud Run is another interesting option that we could consider. It has a free tier/budget that should cover most expenses depending on usage.

[rvalentini]

Had a quick look at Vercel and found this limitation: https://vercel.com/support/articles/does-vercel-support-docker-deployments That's a deal-breaker since the cljdoc service is currently bundled as Docker image, right?

[lread]

@rvalentini I think maybe we can keep options open at this stage.

I'm not sure that deploying a docker image is a hard requirement.
Deploying a jar might be ok too.

That said, we do support running cljdoc locally via a docker image.
So we'd have to think about that use case too.

[lread]

Might be on the obvious side, but probably worth exploring what problems are trying to solve.
Some symptoms:

• It is hard for newcomers to understand ops because there are quite a few moving parts and technologies at play.
• This makes it hard to enlist any help cljdoc ops help from the community.
• Because cljdoc ops is not a full-time job, these tools and technologies are not kept up to date, which makes them harder to maintain. Some of these tools have undergone significant revisions involving such things as file format changes.
• Scripting around these tools is also not kept up to date for the same reason.
• Scripting around these tools is on the complex side. It involves orchestrating all of the moving parts we are managing ourselves.

@martinklepsch might have more to add here.

[martinklepsch]

Ah, I didn't realize Vercel moved away completely from Docker! So yeah, serverless style deployments are probably not a good fit with cljdoc's long-ish startup time.

[rvalentini]

Topic: Move the application state away from the machine running the service

Documentation: https://github.com/cljdoc/cljdoc/blob/master/doc/adr/0013-move-to-sqlite-for-storage.md

[rvalentini]

Are there specific impacts we want to test/explore? For example, I wonder what the impact of fetching/saving to a remote store will be.

* Any latency concerns?

* Any data transfer $ cost concerns?

Might be totally fine, but also interesting to explore.

I gathered some information to answer these questions:

Concerning cost:

• Although we will receive Exoscale sponsoring :) I had a look at their pricing models. We would fit the smallest subscription plan called "Hobbyist", which comes at 38€ / month for Postgres DBaaS. There is no additional cost for networking.
• On DigitalOcean side there might be additional costs for outbound traffic when we reach a certain threshold (1000GiB/month), but currently, our traffic is far below that threshold, so also no concerns here I would say.

Concerning latency:

• Our DigialOcean droplet runs in one of the Amsterdam regions as far as I could see.
• Exoscale offers multiple locations across Europe, of which Frankfurt would be the most favorable for us in this case.
• When we don't touch the DigitalOcean droplet and choose Exoscale Frankfurt region for our DBaaS, I would estimate an additional network latency of 30ms for the round trip, plus some small amount of processing time.
• When we move the droplet also to the Frankfurt region we could further cut down on the network latency if we need to.

So in theory I see no problems, we would have to try out to be sure though.

[lread]

Nice research @rvalentini! Thanks for digging in!

[rvalentini]

Had a great talk with @martinklepsch about how to proceed further on this topic. To get a bit more concrete and explore possible implications, we want to try and connect Postgres as a new storage backend to the application.

Another related question came up though:
What about the SQLite Cache (https://github.com/cljdoc/cljdoc/blob/master/src/cljdoc/util/sqlite_cache.clj)? Does it make sense to also migrate that away from the local disk? As described in the ADR (https://github.com/cljdoc/cljdoc/blob/master/doc/adr/0018-caching-layer.md) the main purpose of it is to avoid network calls. Moving it to an external DBaaS together with the main SQLite DB seems to contradict that intent to some extent.

One of the major benefits of the intended DBaaS migration would be to get rid of the local disk storage, which would open up more options to simplify the infrastructure setup. If the cache remains, and we use both Postgres and SQLite in parallel, the migration might not be so rewarding anymore?

[lread]

Awesome news @rvalentini, thanks for sharing!

Fwiw, I think the local cache is only currently used to cache poms.
Secondary benefits of a cache are sometimes:

• avoiding rate limits
• politeness (not overusing a 3rd party service)
• continuing to operate for cached items when 3rd party service is down

Some popular typical cloud-based caching technologies are Redis and Memcached (or compatible) but I'm not sure if they fit our use case.

[martinklepsch]

I think it'd potentially be ok to wipe the cache on restarts/redeploys. We get the POM for every documentation request and as long as we only do it the first time documentation is loaded and then have it cached for a reasonable while (likely days as we don't deploy all that often) that seems ok to me?

[rvalentini]

Documentation: Current Setup

To understand the current setup better, I tried to sketch the status quo of the infrastructure and what's going on from an operations perspective. Hope this will be helpful for anybody joining the discussion. This is my initial understanding from going over the config files & scripts, so if anything is wrong or missing, please tell and I will update the sketch.

[rvalentini]

[lread]

Awesome! Would be interesting to explicitly draw in where the sqlite databases live.

[rvalentini]

Good idea, that would be the filesystem of the VM running Nomad and the two containers I think. I will add it to the sketch 👍

[lread]

Use case: I am cljdoc developer and need to run locally

Here we explore what this might mean in our new world.

[lread]

Use case: I am library author and want to run locally to preview what my docs will look like

I'm not sure how many folks do this (I do but... yeah).

Current available paths are:

1. from docker
2. from source

[lread]

Use case: I am a cljdoc developer and I'd like to have confidence that my changes will work in a production environment

Today, we get a reasonable level of confidence by running locally under docker.

[lread]

Use case: I am a cljdoc developer and I want to work with a copy of the production db locally.

Currently on the easy side, an sqlite db is simply a file.

[rvalentini]

Topic: Playground Cljdoc environment for experiments

When we want to try out different infrastructure setups or migration possibilities, a second Cljdoc environment might be useful. Maybe we could also use the cljdoc.xyz domain here?

[lread]

Topic: Logging Service

Stick with Sentry? Move to something else? Maybe papertrail?
Inspired by #513