Script to change local-storage to AWS

[jmelati]

Is your feature or enhancement related to a problem? Please describe.

It would be interesting a script to migrate the images that are in a "local storage" installation to an "AWS" type cloud.
I've been using it in a production environment for some time now, and I only felt the need now.
Wish I had the option to migrate.

Describe the solution you'd like

A script who do this migration.

Describe alternatives you've considered

A step-by-step for this.

Additional context

No response

[sojan-official]

ref: https://www.stefanwienert.de/blog/2018/11/05/active-storage-migrate-between-providers-from-local-to-amazon/

[FranciscoJBrito]

@jmelati @sojan-official, This issue is still open? I would like to work on it

[sojan-official]

@FranciscoJBrito please go ahead.

[hiagodotme]

It's cool that this discussion already exists here on Github, I have the same need.

[leonardofilhogm]

I am on the same need. I had an installation done from ctwl script. And now i did a migration to a docker installation using CapRover. I need to migrate the blobs to a minIo bucket.

[dlc-letelier]

Also the same need.

[dlc-letelier]

@FranciscoJBrito , just a comment, maybe can help. I have run the script suggested by @sojan-official , and works fine, the blobs are really imported to the S3 (minIO, in my case). The problem is that, even when it is already imported, Chatwoot still uses the local folder, so if I delete the "/app/storage" folder, I cannot access the data, even when is already in the S3. This happened with the old data in local storage, because the new one is going to the S3.

I discovered that if I change the "service_name" in the "active_storage_blob" to "s3_compatible", works fine.

I think there is something missing here:

https://www.stefanwienert.de/blog/2018/11/05/active-storage-migrate-between-providers-from-local-to-amazon/

Maybe you already resolved this in the #9117 PR. Just an observation, because I am new in rails.

[FranciscoJBrito]

Hi @dlc-letelier 👋🏻.
Yep, the script is for performing migrations, if once you have performed the migrations you want to change the default configuration of active storage, so that the files are stored in another provider other than local, you can do so in the file config/environments/[the environment you want to change], specifically on the line config.active_storage.service which is by default defined as local, if you change it to amazon for example, then your active storage will store the files in AWS S3.

P.S.: Make sure to correctly configure the providers in the file config/storage.yml

[dlc-letelier]

Hi @FranciscoJBrito ,
Thanks for your reply. But the PR will also change the database, or It has to be done manually? Because the Blob entry location has to be change in someway.

[FranciscoJBrito]

@dlc-letelier this script is intended to migrate content that you already have stored in your active storage 'local', to a cloud provider, but it will not change the active storage configuration (by default it will still be 'local'). If you want to change the default configuration you can do it manually in the file config/environments/[desired environment].rb.

[dlc-letelier]

@dlc-letelier this script is intended to migrate content that you already have stored in your active storage 'local', to a cloud provider, but it will not change the active storage configuration (by default it will still be 'local'). If you want to change the default configuration you can do it manually in the file config/environments/[desired environment].rb.

Understood. I'am using Kubernetes, so I do not have to change the environments file, but the values.yaml. But even changing to 's3_compatibles', only the new files are save and read from the S3. The old ones, already migrated to the S3, are taken from the local.
But if there is no solution, I will change the service_name manually in the database.
Thanks.

[hiagodotme]

@FranciscoJBrito I noticed you talked about YAML files, my deployment was with docker. The only configuration file I have is a .ENV

Will it work the same?

[FranciscoJBrito]

Hi @hiagodotme, yes in your .env file you can define the necessary environment variables for the cloud service you want to configure. The attached image shows the storage.yml file located in the config folder at the root of the project.
In your .env file you have to add those environment variables with the corresponding credentials.

[hiagodotme]

Thank you very much! Today I managed to take some time to do my migration.

Just a consideration for @FranciscoJBrito PR #9117 . I had files that were removed, I believe it was because someone had deleted a message with an attachment (as I performed a delete on the storage table and everything went well).

However, a tip to improve the implementation is to ignore files that were not found and in the end create a not_found_log.txt.

Thank you again.

[hiagodotme]

@FranciscoJBrito I also noticed that files received via WhatsApp and email channels were not uploaded to AWS. I placed a comment here displaying the line that prevented the upload:

https://github.com/chatroot/chatroot/pull/9117/commits/5c350348d63af1b1cfd807412cc11fc616044da2

[FranciscoJBrito]

@hiagodotme, perfect, I will make the modification so that you can migrate any type of file as soon as I can