Migration to S3 bucket (Minio) not working. #8657
Labels
bug
Something isn't working
Comments
Logs |
|
Do you mind showing your helm values for the |
|
Ah actually that might not be it. What do you have inside the |
Hi @rissson This is my authentik helm config global:
# -- Common labels for all resources.
additionalLabels:
app: authentik
# Number of old deployment ReplicaSets to retain. The rest will be garbage collected.
revisionHistoryLimit: 3
# Default image used by all authentik components. For GeoIP configuration, see the geoip values below.
image:
# -- If defined, a repository applied to all authentik deployments
repository: ghcr.io/goauthentik/server
# -- Overrides the global authentik whose default is the chart appVersion
tag:
# -- If defined, an image digest applied to all authentik deployments
digest:
# -- If defined, an imagePullPolicy applied to all authentik deployments
pullPolicy: IfNotPresent
# -- Secrets with credentials to pull images from a private registry
imagePullSecrets: []
# -- Annotations for all deployed Deployments
deploymentAnnotations:
reloader.stakater.com/auto: "true"
# -- Annotations for all deployed pods
podAnnotations:
backup.velero.io/backup-volumes: media
# -- Labels for all deployed pods
podLabels: {}
# -- Add Prometheus scrape annotations to all metrics services. This can be used as an alternative to the ServiceMonitors.
addPrometheusAnnotations: false
# -- Toggle and define pod-level security context.
# @default -- `{}` (See [values.yaml])
securityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
# -- Mapping between IP and hostnames that will be injected as entries in the pod's hosts files
hostAliases: []
# - ip: 10.20.30.40
# hostnames:
# - my.hostname
# -- Default priority class for all components
priorityClassName: ""
# -- Default node selector for all components
nodeSelector: []
# -- Default tolerations for all components
tolerations: []
# Default affinity preset for all components
affinity:
# -- Default pod anti-affinity rules. Either: `none`, `soft` or `hard`
podAntiAffinity: soft
# Node affinity rules
nodeAffinity:
# -- Default node affinity rules. Either `none`, `soft` or `hard`
type: hard
# -- Default match expressions for node affinity
matchExpressions: []
# - key: topology.kubernetes.io/zone
# operator: In
# values:
# - zonea
# - zoneb
# -- Default [TopologySpreadConstraints] rules for all components
## Ref: https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/
topologySpreadConstraints: []
# - maxSkew: 1
# topologyKey: topology.kubernetes.io/zone
# whenUnsatisfiable: DoNotSchedule
# -- Deployment strategy for all deployed Deployments
deploymentStrategy: {}
# type: RollingUpdate
# rollingUpdate:
# maxSurge: 25%
# maxUnavailable: 25%
# -- Environment variables to pass to all deployed Deployments. Does not apply to GeoIP
# See configuration options at https://goauthentik.io/docs/installation/configuration/
# @default -- `[]` (See [values.yaml])
env:
- name: AUTHENTIK_STORAGE_MEDIA_BACKEND
value: "s3"
- name: AUTHENTIK_POSTGRESQL__SSLROOTCERT
value: certs/eb-ca-bundle.crt
- name: AUTHENTIK_POSTGRESQL__SSLMODE
value: verify-full
envFrom:
- secretRef:
name: authentik-s3-secret
# -- Additional volumeMounts to all deployed Deployments. Does not apply to GeoIP
# @default -- `[]` (See [values.yaml])
volumeMounts:
- name: media
mountPath: /media
- name: eb-internal-ca-bundle
mountPath: /certs/
readOnly: true
# -- Additional volumes to all deployed Deployments.
# @default -- `[]` (See [values.yaml])
volumes:
- name: media
persistentVolumeClaim:
claimName: authentik-pvc
- name: eb-internal-ca-bundle
secret:
secretName: eb-internal-ca-bundle
defaultMode: 256
items:
- key: ca.crt
path: eb-ca-bundle.crt
## Authentik configuration
authentik:
# -- Log level for server and worker
log_level: info
events:
context_processors:
# -- Path for the GeoIP City database. If the file doesn't exist, GeoIP features are disabled.
geoip: /geoip/GeoLite2-City.mmdb
# -- Path for the GeoIP ASN database. If the file doesn't exist, GeoIP features are disabled.
asn: /geoip/GeoLite2-ASN.mmdb
outposts:
# -- Template used for managed outposts. The following placeholders can be used
# %(type)s - the type of the outpost
# %(version)s - version of your authentik install
# %(build_hash)s - only for beta versions, the build hash of the image
container_image_base: ghcr.io/goauthentik/%(type)s:%(version)s
error_reporting:
# -- This sends anonymous usage-data, stack traces on errors and
# performance data to sentry.beryju.org, and is fully opt-in
enabled: false
# -- This is a string that is sent to sentry with your error reports
environment: k8s
# -- Send PII (Personally identifiable information) data to sentry
send_pii: false
secret_key: XXXXXXX
email:
# -- SMTP Server emails are sent from, fully optional
host: XXXXXXX
# -- SMTP server port
port: XXXXXXX
# -- SMTP credentials, when left empty, no authentication will be done
username: XXXXXXX
# -- SMTP credentials, when left empty, no authentication will be done
password: XXXXXXX
# -- Enable either use_tls or use_ssl, they can't be enabled at the same time.
use_tls: true
# -- Enable either use_tls or use_ssl, they can't be enabled at the same time.
use_ssl: false
# -- Connection timeout
timeout: 30
# -- Email from address, can either be in the format "foo@bar.baz" or "authentik <foo@bar.baz>"
from: XXXXXXX
postgresql:
# -- set the postgresql hostname to talk to
# if unset and .Values.postgresql.enabled == true, will generate the default
# @default -- `{{ .Release.Name }}-postgresql`
host: XXXXXXX
# -- postgresql Database name
# @default -- `authentik`
name: authentik
# -- postgresql Username
# @default -- `authentik`
user: authentik
password: XXXXXXX
port: 5432
redis:
# -- set the redis hostname to talk to
# @default -- `{{ .Release.Name }}-redis-master`
host: XXXXXXX
password: XXXXXXX
blueprints:
# -- List of config maps to mount blueprints from.
# Only keys in the configMap ending with `.yaml` will be discovered and applied.
configMaps: []
# -- List of secrets to mount blueprints from.
# Only keys in the secret ending with `.yaml` will be discovered and applied.
secrets: []
additionalObjects:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: authentik-pvc
namespace: authentik
spec:
storageClassName: pve-zfs-hdd-03
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
- apiVersion: v1
kind: Secret
metadata:
name: authentik-s3-secret
namespace: authentik
data:
AUTHENTIK_STORAGE_MEDIA_S3_ENDPOINT: XXXXXXXX
AUTHENTIK_STORAGE_MEDIA_S3_ACCESS__KEY: XXXXXXX
AUTHENTIK_STORAGE_MEDIA_S3_SECRET__KEY: XXXXXXX
AUTHENTIK_STORAGE_MEDIA_S3_BUCKET__NAME: XXXXXXX
AUTHENTIK_STORAGE_MEDIA_S3_CUSTOM__DOMAIN: XXXXXXX
AUTHENTIK_STORAGE_MEDIA_S3_REGION: XXXXXXX
type: Opaque |
|
The |
|
Could be... In any case these folders are post 2024.2.1 migration. Copied directly from the PVC from which is fully working... Do you suggest deleting it? |
|
Can you try uploading an icon using the web interface (by editing an application for instance), and see if that works and shows up correctly in the UI afterwards? I believe you have uploaded the icons to the PVC directly and set the URL manually instead of uploading them through the interface directly. |
I have tested this. |
|
It's really weird that requests are not sent to S3, because the config looks fine. Could you paste the output of the last line of |
|
I'm seeing the same issue when trying to configure Authentik to use Minio. My instance is deployed with v2024.2.1 of the Helm chart. I ran the command requested and see the following output: {'staticfiles': {'BACKEND': 'django.contrib.staticfiles.storage.StaticFilesStorage'}, 'default': {'BACKEND': 'authentik.root.storages.FileStorage', 'OPTIONS': {'location': PosixPath('media'), 'base_url': '/media/'}}}Shouldn't this have been AUTHENTIK_STORAGE_MEDIA_BACKEND: s3
AUTHENTIK_STORAGE_MEDIA_S3_ENDPOINT: https://s3.example.com
AUTHENTIK_STORAGE_MEDIA_S3_CUSTOM__DOMAIN: s3.example.com/authentik
AUTHENTIK_STORAGE_MEDIA_S3_ACCESS__KEY: ***
AUTHENTIK_STORAGE_MEDIA_S3_SECRET__KEY: ***
AUTHENTIK_STORAGE_MEDIA_S3_BUCKET__NAME: authentikEdit: one other thing I should note is that I can't upload a test file since I have the text box instead of the file input (I assume this is because the |
6 tasks
|
Could you check out the PR above and update as needed? |
Hi @rissson sorry for the delay... I'll test the PR and give you an update. |
|
@rissson At first i got errors due missing files on my S3, i had to create /media folder inside my bucket and move my assets inside, afterwards seems to be fully working 😄
Update: I tried to test file uploading but am getting a HTTP 405 error. I had to disable SSL Verification, enabled debug mode and botocore was complaining about my self signed cert, afterwards this were the errors. |
|
Your S3 server is sending back a |
|
@rissson By what it seems, reading files works fine but the problem is on the POST method... I have checked my Service Account policy and it has full s3:* permissions in the bucket |
|
@mlinares1998 Sure! I bumped to Thank you @rissson! |
Great!!! Maybe the 404's are because i'm using the old path style in my instance. |
@mlinares1998 No, I'm using the subdirectory path. For example, I just uploaded a test image and the URL looks something like this If I were to change my actual domain to AUTHENTIK_STORAGE__MEDIA__BACKEND: s3
AUTHENTIK_STORAGE__MEDIA__S3__ENDPOINT: https://s3.example.com
AUTHENTIK_STORAGE__MEDIA__S3__CUSTOM_DOMAIN: s3.example.com/authentik
AUTHENTIK_STORAGE__MEDIA__S3__ACCESS_KEY: ***
AUTHENTIK_STORAGE__MEDIA__S3__SECRET_KEY: ***
AUTHENTIK_STORAGE__MEDIA__S3__REGION: us-central-1
AUTHENTIK_STORAGE__MEDIA__S3__BUCKET_NAME: authentikNote that the s3 subdomain corresponds to |
That's exactly my same setup 😞 |
|
@mlinares1998 I'm pretty new to Minio, but I think in your case you'd want to set |
I'm pretty new to minio as well 😄 |
|
@mlinares1998 I'm just not sure if Minio accepts S3 API requests to the virtual hostnames, or if it's only for downloading assets. That's why I'd try setting |
|
@mlinares1998 Ah sorry I misunderstood. I'm not sure why you're getting 404s! 🤔 |
Don't worry! Thanks for your help 😄 |
@gabe565 Did the test and indeed it's the reverse proxy which isn't routing authentik PUT requests to minio. |
|
Alright, glad we fixed this issue, thanks for the help! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Hi. I'm trying to migrate from PV /media file storage to a Minio S3 bucket, but am unable to get the files loaded from the bucket after following the tutorial from https://goauthentik.io/docs/installation/storage-s3
I'm using the helm chart with ArgoCD, mounting my custom CA at /certs and setting the Minio keys in a extra secret created as an additionalObject, mounted using secretRef.
After the launch, the variables are correctly set within the pod but i don't have any logs with access attemps to the S3 bucket nor failed SSL verification due the custom CA, even with AUTHENTIK_LOG_LEVEL set as debug, furthermore, removing the /media PVC ends with authentik assets broken due the missing files (404's), adding a replacement via the console put's it on ephemeral /media folder on the pod.
I don't know if i'm doing anything wrong, It's like it doesn't detect the switch to S3. i'll be grateful if you can help me debugging this.
Thanks!
To Reproduce
Follow the steps from https://goauthentik.io/docs/installation/storage-s3 to switch to S3 bucket.
S3 connection values
Expected behavior
Assets being loaded from the bucket rather pod's /media folder
Screenshots
Version and Deployment (please complete the following information):
The text was updated successfully, but these errors were encountered: