Improve development environment defaults and storage setup

[filiperochalopes]

Summary

This PR improves the local development environment by documenting the expected defaults, reducing maintenance overhead from deprecated components, and making the storage layer more practical for development and testing workflows.

Context

The current development setup had a few limitations:

• MinIO is now obsolete in this project context and will no longer be maintained
• the development override file should not be treated as a documented default entrypoint, since it is an override-only file meant for local use
• using an AWS S3 bucket in development prevents some features from being tested properly or consistently
• a more efficient local storage service was needed to replace the previous approach

To address this, the development environment now moves toward a documented default .env template and a backup-based override workflow, while introducing RUSTFS as the preferred local S3-compatible service.

What changed

• added .env.example documenting the expected development defaults for:

  • core services
  • storage configuration
  • optional integrations

• renamed docker-compose.override.yml to docker-compose.override.yml.bak

• updated the repository to ignore the active override file, so local developer-specific overrides are not committed accidentally

• preserved the backup override template so developers can still enable it locally when needed with:

cp docker-compose.override.yml.bak docker-compose.override.yml

• kept the expanded storage-related development setup available in the backup override file

• introduced RUSTFS as the local storage service, replacing the previous dependence on MinIO for active development workflows

Why this approach

This approach keeps the repository cleaner and makes the development workflow more explicit:

• documented defaults live in .env.example
• local override behavior remains available, but is treated as local-only customization
• deprecated infrastructure is no longer presented as the default path
• local storage testing becomes more realistic and less constrained than using AWS S3 directly during development

Benefits

• clearer onboarding for development environments
• less confusion around which files are documented defaults versus local overrides
• fewer accidental commits of machine-specific override changes
• better support for testing storage-dependent features locally
• improved maintainability by moving away from MinIO
• more efficient local S3-compatible behavior through RUSTFS

Notes for reviewers

Developers who need local override behavior can still enable it manually:

cp docker-compose.override.yml.bak docker-compose.override.yml

This preserves flexibility for development without keeping override configuration as a first-class tracked file.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 077e8abe9d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Set storage backend consistently across worker services

api is now pinned to MinIO/RustFS via EXPORT_SERVICE and MINIO_*, but the Celery services still inherit the default backend (core.settings.EXPORT_SERVICE falls back to AWS S3 when unset). That creates a split-brain storage setup where API writes/deletes against MinIO while async tasks run by workers (for example delete_s3_objects triggered via apply_async in core/common/models.py) operate on a different backend, causing export cleanup and other storage-backed tasks to fail or act on the wrong bucket.

Useful? React with 👍 / 👎.

[filiperochalopes]

Solved

[filiperochalopes]

[snyaggarwal]

there are usages of self.client; shouldn't they all move to self.external_client?

[filiperochalopes]

Good point on the naming — external_client was ambiguous. I renamed it to _presign_client to make the intent explicit: it is only used for presigned URL generation, so the signature carries the publicly accessible host. All other operations (upload_file, exists, has_path, delete_objects, etc.) intentionally continue using self.client, since they run server-side inside the container, where only MINIO_ENDPOINT is reachable through the Docker network.

If we move toward using a single external-style variable for everything, the idea is reasonable, but it is not the fastest path for developers because it does not take advantage of Docker networking and the ready-to-use RustFS service. In that setup, we also would not be able to use localhost, since it would not resolve externally. We would need to introduce a proxy plus an /etc/hosts entry, which is a much more complete setup for what is currently intended to be a simple development configuration.

In any case, if you already have MinIO or RustFS running as an externally reachable service on your machine, defining only MINIO_ENDPOINT is enough to configure both self.client and self._presign_client.