Require less setup, be more flexible

[atrauzzi]

The setup for this container is a bit involved. The two steps that I think could be automated with safe defaults in particular are:

• Manual commands have to be run to get to a running state
  -- Solution here would be to have the container optionally check to see if any databases need to be created and then if any migrations need to be run.
• Cannot configure a redis cache prefix

My specific use case is in a quick-draw local development environment. It features things like a local dev smtp trap and locally hosted GUI tools. I was hoping to add Sentry, but there's no way for me to configure it exclusively from a docker-compose file. At some point I'm required to create a database. At some point I'm also required to run the migrations.

[mattrobenolt]

Manual commands have to be run to get to a running state -- Solution here would be to have the container optionally check to see if any databases need to be created and then if any migrations need to be run.

If you want to do this, you could sentry start --upgrade as your startup command. This will trigger the upgrade before starting up the web processes.

Cannot configure a redis cache prefix

Is there a reason you can't just use a different redis/memcached container instead? If you need to do this, you could always use the onbuild tag and shove in your own config that does whatever you want.

[atrauzzi]

sentry start --upgrade would be useful, except that I believe it won't create the database, right? The postgres container only lets you create one database via environment variables and I use that to create the one for my application. An optional environment variable to have the sentry container also issue an SQL CREATE DATABASE statement for the configured DB wouldn't be such a terrible thing. Avoids people having to get into customizing the container.

I'd rather only run one redis container for my local development setup, but admittedly this is less of a problem there. If however I came to run multiple sentry instances in a shared production situation, I'm forced into multiple redis instances.

[mattrobenolt]

Yeah, this use case is outside the scope if what I'm willing to put into this container. If you need to do this, it's pretty easy to just add this behavior into an onbuild have have your own do whatever you want.

In all situations, I'd just recommend running distinct containers for postgres/redis and they have extremely low overhead by running multiple.

Either way, the sentry container should have no business creating and provisioning databases. That's up to the operator on what they want to do there. We just expect you to plug in credentials. This container is not intended to spin up a dev environment and is geared towards production use.

So as-is, this container is sitting neutral and doesn't prohibit or encourage this use case and I don't think it makes sense to encourage it.

We also don't support running multiple Sentry instances using shared resources. Redis is used for much more than just a cache, and there's no way they could play nicely if they're sharing the same Redis instances.

[atrauzzi]

Alright, I'll try to work that all in.

One additional question @mattrobenolt -- Is there a way with onbuild or any other angle that I can populate an initial default user without having to derive a new container?

[mattrobenolt]

In theory, you could do this inside the docker-entrypoint.sh, and override that as well, and call sentry createuser passing all the args needed. If you'd be modifying this anyways to instrument the createdb and whatnot anyways, so could tie in a createuser call too.

[atrauzzi]

I was actually able to push the createdb into the postgres container, they have a .d setup for first-run SQL scripts (handy!).

Maybe a first run .d style setup or something like that could be done for the entrypoint/onbuild?

[mattrobenolt]

Maybe a first run .d style setup or something like that could be done for the entrypoint/onbuild?

I'm not 100% opposed to it, but I'd need to see actual value in doing this out of the box. Meaning, it'd need to support more than just 1 use case since this is more complexity that needs to be maintained and tested which adds more technical burden on us to maintain the project.

So if there's a clear value in doing this where 50+% of users will utilize it, then sure.

[atrauzzi]

I believe I saw a few other tickets here where people were trying to alter the configuration of the server based on the data it stores in the database. I suppose if anyone is of the mind to get changes in there, they'll be best served by being able to issue instructions this way.

I'd think of it as post-migration (fixtures/seeds?). So, that is to say, a lot of people could benefit from it, but you'll know better whether that's too hypothetical and my use case is the only real example of commands being issued.

[mattrobenolt]

I believe I saw a few other tickets here where people were trying to alter the configuration of the server based on the data it stores in the database.

Nothing is coming to my mind about this and this sounds weird to me.

I'd think of it as post-migration. So, that is to say, a lot of people could benefit from it, but you'll know better whether that's too hypothetical and my use case is the only real example of commands being issues.

Until now, not a single person has ever requested something like this, and Sentry has existed long before Docker was a thing. Thousands of users run Sentry today and have survived just fine. :)

Sorry, I don't mean to be argumentative (if I'm coming across this way), it's just important that any additions and complexity are considered fully since I will be the one left to maintain it in the future. This goes with documentation and support from our side.

[atrauzzi]

Oh no, don't worry, this is how we sort it out! ;)

The main thing striking me as different with this series of containers is that you're assuming there's a human on the end of the keyboard to issue some commands all the time (hence this issue originally). And that everything is "well you'll only ever need to do it once".
Docker prides itself on disposable containers, and that usually carries the expectation that they can be scripted/automated. I can't really build around a "service" that won't let me put it in a certain state - or better stated: initial state.

So while sentry has been around for a long time, Docker brings shifts in expectations that aren't going to jive with what has historically been the case. Really, you want people to be able to declaratively set up these containers. Otherwise you're only half-Docker.
Looking back to my example above, even postgres (a tool that could make similar arguments about people needing to manually do setup) accepts that people need to spin up more than one database. But they layer the flexibility. You get an env var, if that's not enough, bind-mount some scripts.

The fact that sentry relies on a few supporting containers doesn't make it unique. What I want to do isn't unusual for containers, it's unusual for a process running on a server that's been set up by hand.

[mattrobenolt]

Yeah, I understand the perspective. I'm just looking for the use case if what would go into a .d folder as some examples.

I can see value in adding a hook to create a user on boot, since that's a normal case. My concern is that this isn't enough to get up and going. You ultimately need to create projects and get a DSN back to use anyways, and there's no CLI way to do this. So at this point, it's not even possible to fully bootstrap without poking into the web UI. Even if you wanted to use the API, you'd need to create an API key through the web UI first.

I'm torn because detecting that a migration needs to run is not a fast operation. If it were quick, I'd happily make update a part of the bootup process by default.

It's also unfortunately not great because you don't want to have concurrent migrations running. So if you were to boot up 10 containers at once, and they all decide to upgrade at the same time, it'll likely be a bad time. Hence why it's an explicit step. In the real world, upgrading is an explicit decision and things are magical.

I'd say that Sentry is not easily comparable to other services just because we need more than 1 container. Sentry is run at a much larger scale than most others though.

I keep pushing for the onbuild container since I still don't understand why what you want can't be encompassed in there? If you want to run scripts on boot, just add them there. They just feel out of place for the base sentry container imo and too fragile for a complex system. Postgres/MySQL are much simpler. I think it'd be hard to have scripts that are supposed to only run once globally, get orchestrated properly in every container booting up at the same time. Sure, in one container, it's not an issue. But when you're spinning up a bunch across multiple machines, something needs to enforce that things don't run concurrently or run many times, etc. And like I said, we don't have quick/simple checks in Sentry to detect if these things need to be done or locking mechanisms to say, prevent multiple migrations from running at once.

And with that said, basically nobody runs the sentry base container, and if they do, they're guaranteed to have a very poor experience. It's much more common to use onbuild anyways to create your own container because of needing to install plugins and their own custom plugins.

[atrauzzi]

Some of this sounds like it could actually filter back up to sentry itself as ideas - without implying it's deficient? Just for this brave new world we are in.

[mattrobenolt]

Yeah, agreed, there are definitely more protections we can do in Sentry to make these operations faster and less error prone, but that's a bit out of the scope here.

With that said, locking upgrade/repair isn't really a challenge. We've never implemented this before since, well, it was never an issue. fwiw, I did this 4 years ago. :) https://github.com/mattrobenolt/django-locking-south

We could easily add a sentry upgrade --exclusive or something akin to that, which would lock down the upgrade process to only one running system wide.

This still doesn't solve the issue with it being faster though.

Either way, this topic is being actively thought about, and I have plans on making this bootstrapping process more automated and a bit less hands on, while taking into account upgrading, etc, but they won't be solved in docker-sentry, they'll be solved at the higher level in Sentry itself, or in my mind, a Sentry cluster management tool. But we'll see how that goes since it's a bit of a far off plan yet and we don't have the resources to execute at the moment.

But I appreciate the feedback, and I'm definitely taking it into consideration when we do get do implementing these things.