Skip to content

feat(local-dev): use garage for s3#1996

Merged
pyphilia merged 4 commits into
mainfrom
use-garage-for-s3
Oct 16, 2025
Merged

feat(local-dev): use garage for s3#1996
pyphilia merged 4 commits into
mainfrom
use-garage-for-s3

Conversation

@spaenleh
Copy link
Copy Markdown
Member

@spaenleh spaenleh commented Oct 3, 2025

In this PR I propose to improve the DX in local by removing the difference between a local-file service an an s3-file service.

Current situation

Currently we support two alternatives for local development:

  • s3-like API with localstack (does not persist unless we use the pro-license)
  • local API with a static file server and direct file write in a volume (supports persistence but not signed urls and other S3 specific things)

This leads to issues when developing locally and adds uncertainty about the developed code and whether it will work as expected when deployed agains S3.

Proposed solution

The proposed solution is to be able to have a working s3 storage in local dev that truly works like s3.
This solution is deployed correctly could replace the localfile solution for the "self hosted" alternative.

The solution uses deuxfleurs/garage a s3 compatible storage solution that also supports replication and is used in production scenarios. It is reliable and production-ready. The fact that it is s3-compatible allows us to continue using the aws-s3 libraries as our storage abstraction.

Implementation and important informations

Setup

The setup for garage has been explained in the readme section. It requires setup of the layout (a concept specific to garage), the bucket, the access key and the permission of the access key on the bucket.

The s3 service can be contacted using s3.garge.localhost from the internal docker network using the links key in the service that wants to connect and from the host network using an /etc/hosts config linking s3.garage.localhost to localhost.

Changes required for current setups

If you already have a dev environment setup, please do the following:

  1. Follow the readme configuration to create the layout, bucket and access key and access key permission in garage
  2. Add your access key and secret to your .env.development file
  3. Change the s3 region to garage in your .env.development file
  4. Ensure you use the new s3.garage.localhost for the url of the file-service if you override it in the env file. If not overriden it should be correctly set by the docker-compose.yml file, eventully rebuild for the change to take effect.

@spaenleh spaenleh self-assigned this Oct 3, 2025
@spaenleh spaenleh requested a review from pyphilia October 13, 2025 09:59
Copy link
Copy Markdown
Contributor

@pyphilia pyphilia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! I've tried and made some comments to improve the docs.

Remove localstack related comments.

Remove ### Data persistence section under Known issues?

Comment thread README.md Outdated
Comment thread README.md Outdated
Comment thread .devcontainer/docker-compose.yml Outdated
Comment thread .devcontainer/docker-compose.yml Outdated
Comment thread .devcontainer/docker-compose.yml Outdated
Comment thread README.md Outdated
Comment thread README.md Outdated
Comment thread README.md Outdated
Comment thread README.md Outdated
Comment thread README.md Outdated
@spaenleh spaenleh requested a review from pyphilia October 15, 2025 12:28
@sonarqubecloud
Copy link
Copy Markdown

@spaenleh
Copy link
Copy Markdown
Member Author

@pyphilia Thank you for your thorough review. I have updated the documentation and texts with your suggestions. You can review one last time and let me know if it is good to merge. I propose that we plan adding the setup for H5P later.

@pyphilia pyphilia added this pull request to the merge queue Oct 16, 2025
Merged via the queue into main with commit 8873b7f Oct 16, 2025
8 checks passed
@pyphilia pyphilia deleted the use-garage-for-s3 branch October 16, 2025 09:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants