Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add mastodon package and module #60788

Open
wants to merge 18 commits into
base: master
from

Conversation

@petabyteboy
Copy link
Contributor

commented May 2, 2019

Motivation for this change

This module can already be used to set up a fully functional mastodon instance, but there is still a lot to do:

  • Discuss if moretea/yarn2nix should be added to nixpkgs again
  • Don't force users to store secrets in nix store
  • Improve documentation of module options
  • Add mastodon user and group ids
  • Add meta information to the mastodon package
  • Investigate if it makes sense to allow enabling and disabling the three services seperately, similar to the kubernetes module

Some things would be nice to have but are not strictly required for a first version in my opinion:

  • Write some tests
  • Investigate if building streaming, sidekiq and web services seperately would be possible and advantageous
  • Add more advanced options, i.e. the number of sidekiq threads, support for S3 storage backend, ...
  • Package mastodon tools like tootctl in a way that makes them easy to use

Any feedback is appreciated :-)

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Assured whether relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@petabyteboy petabyteboy force-pushed the petabyteboy:feature/mastodon branch 7 times, most recently from c750ceb to d8e0488 May 2, 2019

@petabyteboy petabyteboy marked this pull request as ready for review May 2, 2019

@petabyteboy petabyteboy requested a review from Infinisil as a code owner May 2, 2019

@petabyteboy petabyteboy force-pushed the petabyteboy:feature/mastodon branch from 4c5bd7e to 4f017df May 2, 2019

@petabyteboy

This comment has been minimized.

Copy link
Contributor Author

commented May 2, 2019

Of course we are facing the same IFD issue with yarn2nix as with riot-desktop and #59111.
It seems like the only solution for now is including yarn.nix and package.json in nixpkgs.

@petabyteboy petabyteboy force-pushed the petabyteboy:feature/mastodon branch 2 times, most recently from 1ebe235 to c9b430f May 2, 2019

@petabyteboy

This comment has been minimized.

Copy link
Contributor Author

commented May 2, 2019

Unfortunately it seems like this is a dead end:
moretea/yarn2nix uses builtins.fetchGit, because the yarn.lock file does not contain hashes for git dependencies. builtins.fetchGit doesn't work on Hydra/OfBorg, because it runs at eval-time in restricted mode and network access is not allowed.

I will try to use https://github.com/Profpatsch/yarn2nix tomorrow, which hashes the git dependencies when creating the nix expression, and then uses pkgs.fetchgit. Since we have to include a pregenerated Nix expression for the dependencies anyways (otherwise we get IFD problems), this is a small loss.

@alyssais

This comment has been minimized.

Copy link
Member

commented May 3, 2019

@petabyteboy

This comment has been minimized.

Copy link
Contributor Author

commented May 3, 2019

Can I do ...? @GrahamcOfBorg eval
As I understand it, you should be able to eval, but not build, unless you're a known or trusted user, in OfBorg terminology.

That's what I concluded too, but as it says in the OfBorg readme, there is no reason to call eval manually since it happens automatically.

@petabyteboy

This comment has been minimized.

Copy link
Contributor Author

commented May 3, 2019

After trying to integrate profpatsch/yarn2nix into nixpkgs, I give up on that. There are multiple broken Haskell packages required for profpatsch/yarn2nix to run.

My new plan is to add support for Hydra-enabled git dependencies to moretea/yarn2nix.

@petabyteboy petabyteboy force-pushed the petabyteboy:feature/mastodon branch from c9b430f to fd487c7 May 3, 2019

@petabyteboy petabyteboy force-pushed the petabyteboy:feature/mastodon branch from fd487c7 to d549b7c May 3, 2019

@petabyteboy

This comment has been minimized.

Copy link
Contributor Author

commented May 3, 2019

Hooray /o/

petabyteboy added some commits May 8, 2019

nixos/mastodon: disable nginx by default
add some documentation on what the nginx is configured to do

@petabyteboy petabyteboy force-pushed the petabyteboy:feature/mastodon branch from 34248d1 to 554caff Jun 21, 2019

@matthiasbeyer

This comment has been minimized.

Copy link
Contributor

commented Jun 29, 2019

Any progress?

@petabyteboy petabyteboy referenced this pull request Jul 9, 2019
@ashkitten

This comment has been minimized.

Copy link
Contributor

commented Jul 20, 2019

is there no documentation on how to update the package?

@ashkitten

This comment has been minimized.

Copy link
Contributor

commented Jul 20, 2019

i played around with this myself today, and i believe it needs quite a few modifications before it's actually usable:

  • the nginx config that comes with it does not use recommendedProxySettings, which is required so that mastodon won't just continually redirect to https://127.0.0.1
  • additionally, nginx needs to have SSL, as recent versions of mastodon do not support http in production.
  • it's not clear that you will need to set up postgresql separately
  • by default it uses postgresql via a local ip instead of the unix socket, which requires that a password be set
  • it does not set up the database for you and there is no indication that you will have to set up the database yourself
  • a lot of options are required to use the service at all, which are not actually necessary - we can make .env settings optional.
  • putting each secret in a separate file is very confusing, especially since they are all lumped into a single file afterwards anyway
  • when changing one of the secret variables, the secrets file was not regenerated. i couldn't figure out how to get it to regenerate, and ended up having to edit the secrets file manually
  • i set it up with my personal fork of mastodon, which ended up being a real pain to figure out how to update dependencies. having to include both the package.json and yarn.lock as well as generate yarn.nix and gemset.nix was a nightmare, especially since i didn't know i had to apply mastodon-nix.patch before using bundix and yarn2nix to generate those files. needs better workflow and documentation.
@petabyteboy

This comment has been minimized.

Copy link
Contributor Author

commented Jul 20, 2019

Thanks for the review. Do you have any idea how we can improve this?

@aanderse

This comment has been minimized.

Copy link
Contributor

commented Jul 20, 2019

As far as the database goes maybe you can lump all database options under a database attribute like many other services do and then eventually take advantage of this when it gets done.

FYI there are services.postgresql.enensureDatabases and services.postgresql.ensureUsers options in master that will provision a database and database user for you, which combine nicely with a database.createLocally option. Take a look at zabbix in master for an example of working with databases. Bonus points for supporting multiple database types if the application allows it 😆

@ashkitten

This comment has been minimized.

Copy link
Contributor

commented Jul 20, 2019

for the secrets, i don't see any point in having some of the env variables separate from the rest. i think we should just have an envFile option, which also solves the issue of the user wanting to specify things that aren't configurable in this module. however, we still need the domain name for configuring nginx if we still want to do that. i suppose we can keep the domain as a configuration option in the module, but we have to decide whether that will set the domain for mastodon or just nginx.

@petabyteboy

This comment has been minimized.

Copy link
Contributor Author

commented Jul 20, 2019

for the secrets, i don't see any point in having some of the env variables separate from the rest. i think we should just have an envFile option, which also solves the issue of the user wanting to specify things that aren't configurable in this module. however, we still need the domain name for configuring nginx if > we still want to do that. i suppose we can keep the domain as a configuration option in the module, but we have to decide whether that will set the domain for mastodon or just nginx.

I disagree about the env file: The env file contains all configuration options. Managing those configuration options is the purpose of the module after all. Managing those values in nix makes setting and overriding options much easier.
For the env file not always being regenerated, this can be solved by documentation or maybe we can adjust the systemd units to regenerate the configuration on every start of the mastodon service.

@ashkitten

This comment has been minimized.

Copy link
Contributor

commented Jul 21, 2019

in that case we definitely need an extraEnv and extraEnvFile option in the module to make it possible to specify other variables and possibly other secrets.

@petabyteboy

This comment has been minimized.

Copy link
Contributor Author

commented Jul 21, 2019

I will update to 2.9.2 and implement the suggestions (changes for database options, extraEnv) soon :-)

@petabyteboy

This comment has been minimized.

Copy link
Contributor Author

commented Jul 21, 2019

i played around with this myself today, and i believe it needs quite a few modifications before it's actually usable:

  • the nginx config that comes with it does not use recommendedProxySettings, which is required so that mastodon won't just continually redirect to https://127.0.0.1
  • additionally, nginx needs to have SSL, as recent versions of mastodon do not support http in production.
  • it's not clear that you will need to set up postgresql separately
  • by default it uses postgresql via a local ip instead of the unix socket, which requires that a password be set
  • it does not set up the database for you and there is no indication that you will have to set up the database yourself
  • a lot of options are required to use the service at all, which are not actually necessary - we can make .env settings optional.
  • putting each secret in a separate file is very confusing, especially since they are all lumped into a single file afterwards anyway
  • when changing one of the secret variables, the secrets file was not regenerated. i couldn't figure out how to get it to regenerate, and ended up having to edit the secrets file manually
  • i set it up with my personal fork of mastodon, which ended up being a real pain to figure out how to update dependencies. having to include both the package.json and yarn.lock as well as generate yarn.nix and gemset.nix was a nightmare, especially since i didn't know i had to apply mastodon-nix.patch before using bundix and yarn2nix to generate those files. needs better workflow and documentation.

On updating to a new version or using a forked version, I think we might be able to upstream some of our changes if upstream allows it. Also note that some of the features in the nix tooling are not merged yet (moretea/yarn2nix#100, manveru/bundix#48), which makes it difficult to generate yarn.nix and gemset.nix.

@ashkitten

This comment has been minimized.

Copy link
Contributor

commented Jul 21, 2019

it would be really great if we could generate yarn.nix and gemset.nix on the fly, and then we would be able to simply specify what repository we want to use as a module option.

@petabyteboy

This comment has been minimized.

Copy link
Contributor Author

commented Jul 21, 2019

it would be really great if we could generate yarn.nix and gemset.nix on the fly, and then we would be able to simply specify what repository we want to use as a module option.

Unfortunately this is not possible due to Hydra restrictions.

@ashkitten

This comment has been minimized.

Copy link
Contributor

commented Jul 21, 2019

it would be really great if we could generate yarn.nix and gemset.nix on the fly, and then we would be able to simply specify what repository we want to use as a module option.

Unfortunately this is not possible due to Hydra restrictions.

i'm unclear, what is hydra unable to do?

@petabyteboy

This comment has been minimized.

Copy link
Contributor Author

commented Jul 21, 2019

yarn2nix natively supports generating yarn.nix on the fly, but it requires IFD (import from derivation) functionality, which is disabled in Hydra builds. When you import .nix files from previous build outputs, that's IFD. All .nix files interpreted by Hydra need to be part of nixpkgs.

@petabyteboy

This comment has been minimized.

Copy link
Contributor Author

commented Jul 21, 2019

Another restriction with gemset.nix is that the tool generating it requires an internet connection, because the Gemfile/Gemfile.lock doesn't contain usable hashes. So while yarn.nix contains the same information as yarn.lock (and can be generated without an internet connection), gemset.nix contains additional information pulled from the internet (the hashes of all dependencies).
Even for yarn.nix there is another problem: Git dependencies don't have a hash in yarn.lock, the builtin fetchGit function doesn't work on Hydra, and fetchgit requires a hash again which is not contained in the mastodon sources.

I have spent a lot of time thinking through this and we will always need a manual step outside the build to prepare the mastodon source. (but of course it could be a lot easier if all the tooling like bundix and yarn2nix was available in nixpkgs with the correct patches and the changes needed to make the mastodon depnedencies processable by our tooling could be minimalized).

@ashkitten

This comment has been minimized.

Copy link
Contributor

commented Jul 21, 2019

hmm, that's unfortunate. if it can't be generated during build, perhaps we should have a maintainer script at least which prepares everything to be built

@petabyteboy

This comment has been minimized.

Copy link
Contributor Author

commented Jul 21, 2019

I have another idea on how we can make it easier to use a custom mastodon source. When overriding the mastodon source, the package is built locally, so the Hydra restrictions don't apply.
There is still the problem with gemset.nix and yarn.nix containing extra information that needs to be fetched from the internet, but this could be worked around with a fixed-output derivation, although I think someone was discouraging this.

@ashkitten

This comment has been minimized.

Copy link
Contributor

commented Jul 21, 2019

not sure how you mean to have gemset.nix and yarn.nix generated during build, but only if the source is overridden

@ashkitten

This comment has been minimized.

Copy link
Contributor

commented Jul 21, 2019

i would feel bad about merging this if there isn't first-class support for custom forks of mastodon. to be honest, if we can't provide a module option to let the user easily specify a custom source i'd rather provide this as an overlay in a separate repository

@spacekookie

This comment has been minimized.

Copy link
Contributor

commented Jul 21, 2019

@ashkitten:

  • it's not clear that you will need to set up postgresql separately

Isn't that default behaviour for most modules? There's other modules (such as quassel, I believe) where you need to make sure postgres is explicitly configured to make it work.

Apart from that, I don't feel this is a bad thing, especially considering that database setup is something that people are gonna have feelings about. In fact, I'd go as far as saying that it'd make me feel a bit weird if the mastodon module did this for me.

@aanderse

This comment has been minimized.

Copy link
Contributor

commented Jul 21, 2019

I think one goal of NixOS module authors should always be to make as little configuration work for the user as possible so that a normal user of a software who just doesn't care how things are implemented can type services.foo.enable = true; and not too much more to have a working service.

I think another goal should be to allow advanced users a reasonable amount of customization so they aren't locked in with generic options that the module author wrote.

Many modules automatically provision databases and most of those have a database.createLocally option which defaults totrue. Advanced users should feel free to set that to false because they know what they want and how to achieve it.

@petabyteboy

This comment has been minimized.

Copy link
Contributor Author

commented Jul 22, 2019

I have never used such a module that creates a database for me by default. Thanks for explaining how some modules do it @aanderse! I see how this would be a good addition, but is it actually required for all new modules? I know a number of modules that don't set up all required services. If it is not a requirement, I would like to add support for database.createLocally later.

@aanderse

This comment has been minimized.

Copy link
Contributor

commented Jul 22, 2019

@petabyteboy Definitely not a requirement! Just a nice to have 😄

@spacekookie Good to know about quassel. If I find the time I'll try and add a database option to it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.