Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for integrated MariaDB and PostgreSQL #223

Merged
merged 4 commits into from
Apr 1, 2024

Conversation

djjudas21
Copy link
Contributor

@djjudas21 djjudas21 commented Mar 27, 2024

This PR adds basic support for deploying MariaDB or PostgreSQL as part of an Authelia deployment.

Typically the dependency subcharts would be integrated under the top level mariadb: or postgresql: keys, where the config can be inherited by the subcharts, and the main app (i.e. Authelia) would also draw its config from those keys. However the database config in the chart is under storage.mysql and storage.postgres and changing this would be a breaking change.

So I have added two booleans storage.mysql.deploy and storage.postgres.deploy which enable the user to deploy the Bitnami subcharts. This does not affect any existing Authelia deployments. At the moment the user has to copy a few settings (e.g. credentials) from the mariadb section to the storage.mysql section.

In future we could consider a tighter integration, but it would be a breaking change, so let's go with this for now.

Happy to discuss or tweak this, including adding a Redis subchart.

Relevant to #172 and #89 - but won't actually close them until Redis support is added.

Summary by CodeRabbit

  • New Features
    • Introduced deployment options for MariaDB and PostgreSQL instances within the application.

Copy link

coderabbitai bot commented Mar 27, 2024

Important

Auto Review Skipped

Auto reviews are disabled on base/target branches other than the default branch. Please add the base/target branch pattern to the list of additional branches to be reviewed in the settings.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository.

To trigger a single review, invoke the @coderabbitai review command.

Walkthrough

This update introduces the functionality to deploy MariaDB and PostgreSQL instances directly via configuration options in Authelia's Helm chart. This enhancement simplifies the process of setting up databases for Authelia, making it more convenient for users to configure their instances according to their preferences and requirements.

Changes

Files Change Summary
charts/authelia/README.md Added configuration options for deploying MariaDB and PostgreSQL instances.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • The JSON schema for the configuration file is available here.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

CodeRabbit Discord Community

Join our Discord Community to get help, request features, and share feedback.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between 5eac62d and 4978c74.
Files ignored due to path filters (2)
  • charts/authelia/Chart.yaml is excluded by: !**/*.yaml
  • charts/authelia/values.yaml is excluded by: !**/*.yaml
Files selected for processing (1)
  • charts/authelia/README.md (1 hunks)
Additional comments: 2
charts/authelia/README.md (2)
  • 144-144: The addition of the storage.mysql.deploy option to deploy a MariaDB instance is clear and aligns with the PR objectives. However, it might be beneficial to expand this section to briefly explain the scenarios in which a user might want to enable this option, emphasizing its optional nature and potential use cases.
  • 146-146: The introduction of the storage.postgres.deploy option for deploying a PostgreSQL instance is well-documented. To further assist users, consider providing a brief explanation of when enabling this option might be advantageous, highlighting its optional nature and potential benefits for certain deployment scenarios.

Copy link
Member

@james-d-elliott james-d-elliott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please rebase on #215 and make a merge request to that branch instead.

@djjudas21 djjudas21 changed the base branch from master to feat-4.38.0 March 31, 2024 18:41
Signed-off-by: Jonathan <me@jonathangazeley.com>
Signed-off-by: Jonathan <me@jonathangazeley.com>
Copy link
Member

@james-d-elliott james-d-elliott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, not sure about the deploy option and how it exactly works.

@@ -1305,6 +1305,7 @@ configMap:
##
mysql:
enabled: false
deploy: false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this option do? It seems ineffectual.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The deploy option toggles whether the bitnami/mariadb subchart is brought into play, and this condition is set in Chart.yaml. If the mariadb chart is activated, it will then consume the mariadb section of values.yaml.

Typically this condition would operate on mysql.enabled but I wanted to avoid breaking existing users of the chart, where people might want to enable mysql support but keep using their existing instance of mysql rather than have the chart deploy a new instance. So I created a new boolean for this.

Usually the mysql and mariadb sections would be merged into one section, taking the name of the subchart (so mariadb) but this would also break existing users so I chose not to refactor this at the moment. It's something we could talk about for a breaking release, though?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah nice, didn't know abotu that interaction and didn't notice it! LGTM!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also as far as that change, I'm not entirely sure I care about a breaking change for 0.9.0, it's LONG been advertised it will be and v0 rules apply. See #140 and the target PR for more context. I think we can merge as is and decide later what we want to do. I like giving users the option to deploy it or use existing.

As far as MySQL/MariaDB users they're masochists who love things to break anyway. ;)

@@ -1350,6 +1351,7 @@ configMap:
##
postgres:
enabled: false
deploy: false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise here?

@james-d-elliott james-d-elliott self-requested a review April 1, 2024 08:59
@james-d-elliott james-d-elliott merged commit 7bbbb37 into authelia:feat-4.38.0 Apr 1, 2024
4 checks passed
@djjudas21
Copy link
Contributor Author

Great, thanks. As Redis is also a dependency I plan to follow this up with another near-identical PR to pull in the Redis helm chart, which means users will then be able to deploy a full working Authelia stack from one helm chart. I'll get on that a bit later today.

@james-d-elliott
Copy link
Member

Nice! Thanks a lot, I've been stretched thin.

@james-d-elliott
Copy link
Member

Looks like this included a bug making the chart unable to be packaged. I've reverted it for now in bbbd05d. There's also this comment which I've failed to decode: #220 (comment)

We'd welcome these changes in another PR. You should be able to just git revert bbbd05d75c0cd1b2cec0e4d3bd669488d805f7e2 && git checkout origin/master -- .buildkite/pipeline.yaml && git commit -C HEAD --amend to get back to the previous state with PostgreSQL, MariaDB, and Redis.

@djjudas21
Copy link
Contributor Author

OK, I'll have a look at this and see if I can track down what went wrong.

I saw #220 (comment) too and also didn't understand it. I guess it might be to do with what I mentioned a little while ago, about having to set some of the values in two places to make the new integrations work. If you're not concerned about breaking changes, I'll try and integrate those more smoothly.

@james-d-elliott
Copy link
Member

james-d-elliott commented Apr 20, 2024

OK, I'll have a look at this and see if I can track down what went wrong.

I saw #220 (comment) too and also didn't understand it. I guess it might be to do with what I mentioned a little while ago, about having to set some of the values in two places to make the new integrations work. If you're not concerned about breaking changes, I'll try and integrate those more smoothly.

We're in v0 (every minor bump should be expected as breaking), and once we make v1 I'll be fairly happy to bump major versions whenever necessary. The maintenance cycle of the chart will likely never perfectly marry-up to Authelia itself and we have much more freedom.

Only real concern is deploying these dependencies by default should not be the case, it should be explicit. Otherwise as long as the breaking change is documented and makes sense (I've done my best to document the other ones, if you notice any missed ones I'd appreciate a heads up or for those to be fixed too).

@djjudas21
Copy link
Contributor Author

@james-d-elliott I've just got the bandwidth to look at this again.

Looks like this included a bug making the chart unable to be packaged.

You got a link to the actual problem? I couldn't see any failed runs in Github Actions - thanks.

@djjudas21
Copy link
Contributor Author

OK, I've figured it out. When deps are packaged with a Helm chart, there is a helm dep lock or helm dep update step that resolves the dependencies, updates a lockfile and packages the subcharts too. However in your .gitignore, all of these artifacts are ignored (which is quite common)

# .gitignore

# Helm
charts/*/charts
charts/*/Chart.lock

So the usual way of handling this is for your CI/CD pipeline to fetch these deps when the chart is packaged. You will need to execute helm repo add bitnami https://charts.bitnami.com/bitnami in your pipeline immediately before any of the cr steps, i.e. just before here: https://github.com/authelia/chartrepo/blob/master/.buildkite/pipeline.yaml#L19

Then the subcharts will be included in the package you build and publish, but not your git repo.

@james-d-elliott
Copy link
Member

@djjudas21
Copy link
Contributor Author

OK. Can you try explicitly running helm dep lock after adding the Bitnami repo?

@james-d-elliott
Copy link
Member

james-d-elliott commented Apr 29, 2024

That command doesn't exist, looks like it should be helm dependency build but that has the same issues.. also I can reliably replicate this locally with helm 3.14.3 (without using chart-releaser).

@james-d-elliott
Copy link
Member

james-d-elliott commented Apr 29, 2024

helm repo add bitnami https://charts.bitnami.com/bitnami
helm dependency build charts/authelia
helm package charts/authelia
Error: found in Chart.yaml, but missing in charts/ directory: postgresql, mariadb, redis

@djjudas21
Copy link
Contributor Author

Interesting, I'm also on Helm 3.14.3 and it works for me:

[jonathan@poseidon authelia-chartrepo]$ helm dependency build charts/authelia
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "immich" chart repository
...Successfully got an update from the "beluga-cloud" chart repository
...Unable to get an update from the "openebs" chart repository (https://openebs.github.io/charts):
	failed to fetch https://openebs.github.io/charts/index.yaml : 404 Not Found
...Successfully got an update from the "rook-release" chart repository
...Successfully got an update from the "node-feature-discovery" chart repository
...Successfully got an update from the "portainer" chart repository
...Successfully got an update from the "eugen" chart repository
...Successfully got an update from the "authelia" chart repository
...Successfully got an update from the "vikunja" chart repository
...Successfully got an update from the "jetstack" chart repository
...Successfully got an update from the "democratic-csi" chart repository
...Successfully got an update from the "harbor" chart repository
...Successfully got an update from the "kongz" chart repository
...Successfully got an update from the "vmware-tanzu" chart repository
...Successfully got an update from the "djjudas21" chart repository
...Successfully got an update from the "sosivio" chart repository
...Successfully got an update from the "kokuwa" chart repository
...Successfully got an update from the "camerahub" chart repository
...Successfully got an update from the "actions-runner-controller" chart repository
...Successfully got an update from the "k8s-home-lab" chart repository
...Successfully got an update from the "cloudhippie" chart repository
...Successfully got an update from the "dex" chart repository
...Successfully got an update from the "fairwinds-stable" chart repository
...Successfully got an update from the "runix" chart repository
...Successfully got an update from the "percona" chart repository
...Successfully got an update from the "deliveryhero" chart repository
...Successfully got an update from the "intel" chart repository
...Successfully got an update from the "oauth2-proxy" chart repository
...Successfully got an update from the "nextcloud" chart repository
...Successfully got an update from the "influxdata" chart repository
...Successfully got an update from the "node-exporter-textfiles" chart repository
...Successfully got an update from the "adfinis" chart repository
...Successfully got an update from the "mattermost" chart repository
...Successfully got an update from the "kasten" chart repository
...Successfully got an update from the "nicholaswilde" chart repository
...Successfully got an update from the "ananace-charts" chart repository
...Successfully got an update from the "secureCodeBox" chart repository
...Successfully got an update from the "ntppool" chart repository
...Successfully got an update from the "gabe565" chart repository
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "prometheus-community" chart repository
...Successfully got an update from the "bitnami" chart repository
...Successfully got an update from the "truecharts" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 3 charts
Downloading postgresql from repo https://charts.bitnami.com/bitnami
Downloading mariadb from repo https://charts.bitnami.com/bitnami
Downloading redis from repo https://charts.bitnami.com/bitnami
Deleting outdated charts

It is supposed to update all its chart repos when you run helm dependency build, but can you also try running helm repo update bitnami and helm search repo mariadb to make sure it can find the bitnami/mariadb chart?

@james-d-elliott
Copy link
Member

Yep, it finds all the charts, and it downloads them fine. What does the packaged chart look like when you extract it?

@djjudas21
Copy link
Contributor Author

So weird, I'm running into the same error too:

[jonathan@latitude authelia]$ pwd
/home/jonathan/git/authelia-chartrepo/charts/authelia

[jonathan@latitude authelia]$ helm dependency update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "awx-operator" chart repository
...Successfully got an update from the "tx-smtp-relay-helm" chart repository
...Successfully got an update from the "node-feature-discovery" chart repository
...Unable to get an update from the "openebs" chart repository (https://openebs.github.io/charts):
	failed to fetch https://openebs.github.io/charts/index.yaml : 404 Not Found
...Successfully got an update from the "immich" chart repository
...Successfully got an update from the "sosivio" chart repository
...Successfully got an update from the "authelia" chart repository
...Successfully got an update from the "democratic-csi" chart repository
...Successfully got an update from the "camerahub" chart repository
...Successfully got an update from the "actions-runner-controller" chart repository
...Successfully got an update from the "portainer" chart repository
...Successfully got an update from the "jetstack" chart repository
...Successfully got an update from the "beluga-cloud" chart repository
...Successfully got an update from the "runix" chart repository
...Successfully got an update from the "intel" chart repository
...Successfully got an update from the "vikunja" chart repository
...Successfully got an update from the "oauth2-proxy" chart repository
...Successfully got an update from the "k8s-home-lab" chart repository
...Successfully got an update from the "vmware-tanzu" chart repository
...Successfully got an update from the "t3n" chart repository
...Successfully got an update from the "deliveryhero" chart repository
...Successfully got an update from the "rook-release" chart repository
...Successfully got an update from the "harbor" chart repository
...Successfully got an update from the "nextcloud" chart repository
...Successfully got an update from the "influxdata" chart repository
...Successfully got an update from the "djjudas21" chart repository
...Successfully got an update from the "mattermost" chart repository
...Successfully got an update from the "kasten" chart repository
...Successfully got an update from the "ananace-charts" chart repository
...Successfully got an update from the "adfinis" chart repository
...Successfully got an update from the "gitlab" chart repository
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "secureCodeBox" chart repository
...Successfully got an update from the "k8s-at-home" chart repository
...Successfully got an update from the "gabe565" chart repository
...Successfully got an update from the "fairwinds-stable" chart repository
...Successfully got an update from the "bitnami" chart repository
...Successfully got an update from the "prometheus-community" chart repository
...Successfully got an update from the "truecharts" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 3 charts
Downloading postgresql from repo https://charts.bitnami.com/bitnami
Downloading redis from repo https://charts.bitnami.com/bitnami
Downloading mariadb from repo https://charts.bitnami.com/bitnami
Deleting outdated charts

[jonathan@latitude authelia]$ ls charts/
mariadb-18.0.2.tgz  postgresql-15.2.5.tgz  redis-19.1.3.tgz

[jonathan@latitude authelia]$ helm package .
Error: found in Chart.yaml, but missing in charts/ directory: postgresql, redis, mariadb

So we can deduce that the problem is not with dependency resolving, but with packaging behaviour. I've had a look in Helm's issue tracker and I can't see anything relevant there. This is definitely broken behaviour so I'm going to report it as a bug in Helm.

@djjudas21
Copy link
Contributor Author

Reported helm/helm#12992

@james-d-elliott
Copy link
Member

it's the *.tgz line in ./charts/authelia/.helmignore

@djjudas21
Copy link
Contributor Author

it's the *.tgz line in ./charts/authelia/.helmignore

Good catch!

Are you able to re-merge this PR now, or does it need rebasing etc? Thanks

@djjudas21
Copy link
Contributor Author

@james-d-elliott what do you need me to do to progress this? Which branch is the right one to develop against now, v0.9.0-beta3? I'm happy to rework this PR and #232 against the latest branch. Would be great to get #172 and #225 closed 😀

@james-d-elliott
Copy link
Member

master is the correct branch, I don't really have time to work on it but would welcome the PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants