Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow configure services for individual attachments #34935

Merged

Conversation

DmitryTsepelev
Copy link
Contributor

@DmitryTsepelev DmitryTsepelev commented Jan 14, 2019

Idea is taken from this comment

This change allows to configure custom services for individual attachments in the following way:

class User < ActiveRecord::Base
  has_one_attached :avatar, service: :s3
end

If specified service is not properly configured - has_one_attached would throw an error. Since service_name is stored inside active_storage_blobs table it's also possible that misconfiguration will appear in the realtime - for such cases, there is a special configuration option Rails.application.configuration.active_storage.service_misconfiguration_behavior, which can take the following values: :log (default), :fallback_to_default and :raise.

In order to migrate existing apps users would have to re-run rails active_storage:install and rails db:migrate to add a new migration, suggestions are welcome :)

On the side note, I've moved require "database/setup" to the beginning of the test helper to make it look more like a real boot process - migrations are executed before app starts.

@DmitryTsepelev DmitryTsepelev force-pushed the configure-store-per-attachment branch 2 times, most recently from c9ea062 to b3d16cc Compare Jan 14, 2019
Copy link
Contributor

@palkan palkan left a comment

In order to migrate existing apps users would have to re-run rails active_storage:install and rails db:migrate to add a new migration, suggestions are welcome :)

Can we make the presence of service_name column optional?

Right now we check for it in the macroses (https://github.com/rails/rails/pull/34935/files#diff-fc054319f93d887d4427a7a728670ba9R159), but we still rely on service_name column in Blob#service.

What about extracting this feature check (say, ActiveStorage.per_blob_service_supported?) and re-use in Blob#service to avoid calling #service_name?

configs = Rails.configuration.active_storage.service_configurations

begin
ActiveStorage::Service.configure service_name, configs
Copy link
Contributor

@palkan palkan Jan 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would build a new service instance for each blob–we need some kind of memoization here (ServiceRegistry.fetch?). That could also allow us to move exception handling logic there and make #service method much more readable:

def service
  ActiveStorage::ServiceRegistry.fetch(service_name, self.class.service)
end

Copy link
Contributor Author

@DmitryTsepelev DmitryTsepelev Jan 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added memoization, but it looks hard to move exception handling to the same place cause error messages are different enough

@DmitryTsepelev DmitryTsepelev force-pushed the configure-store-per-attachment branch 2 times, most recently from 398c000 to f5d5fa6 Compare Jan 15, 2019
`RuntimeError`.

NOTE: if you've initialized ActiveStorage before this feature was shipped, Rails would
ask you to generate and execute one more migration (`rails active_storage:install && rails db:migrate`).
Copy link
Contributor

@bogdanvlviv bogdanvlviv Jan 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove this NOTE when following the approach described in #34935 (comment)

Copy link
Contributor Author

@DmitryTsepelev DmitryTsepelev Jan 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✔️

@DmitryTsepelev DmitryTsepelev force-pushed the configure-store-per-attachment branch 4 times, most recently from 717ee50 to b19d802 Compare Jan 18, 2019
@rails-bot rails-bot bot added the actiontext label Jan 18, 2019
@DmitryTsepelev DmitryTsepelev force-pushed the configure-store-per-attachment branch 3 times, most recently from 43b8441 to e40eb8f Compare Jan 20, 2019
@DmitryTsepelev
Copy link
Contributor Author

DmitryTsepelev commented Jan 21, 2019

r? @georgeclaghorn

@abhchand
Copy link
Contributor

abhchand commented Feb 18, 2019

Hey all, just came across this PR while browsing other activestorage PRs.

I just recently opened #35290 that adds the ability to configure attachments.

I don't want to get ahead of myself and assume that change will be accepted. But if it is accepted I wanted to suggest that we could use that same mechanism here when configuring the service.

class User < ApplicationRecord
  has_one_attached :avatar do |attachment|
    attachment.service :s3
  end
end

Just a thought. Thanks!

@DmitryTsepelev
Copy link
Contributor Author

DmitryTsepelev commented Feb 18, 2019

Hi @abhchand, thanks for letting me know! I would add it to my PR if yours will be accepted earlier

@iggant
Copy link

iggant commented Apr 2, 2019

There is no activity on this PR, but will be nice to have this feature in Rails 6.0
@DmitryTsepelev can you please resolve conflicts and maybe there will be any chance to review this again?

@DmitryTsepelev
Copy link
Contributor Author

DmitryTsepelev commented Apr 2, 2019

Hi @iggant, thanks for the support, I'd like to have it merged too :) The only conflict is in the changelog so it shouldn't be a blocker for a review. I believe core team members are busy polishing release candidates

@iggant
Copy link

iggant commented Apr 4, 2019

@bogdanvlviv Can you please review this PR again, will be very nice to have this feature in new major Rails release

@lordjavac
Copy link

lordjavac commented Apr 5, 2019

Lack of this feature is the one thing preventing me from moving to ActiveStorage. I'd love to see this get implemented soon. Thanks for the hard work!

@DmitryTsepelev
Copy link
Contributor Author

DmitryTsepelev commented Apr 11, 2019

Hi @georgeclaghorn, is there a chance this PR can fit into 6.0.0? 🙂

@georgeclaghorn
Copy link
Contributor

georgeclaghorn commented Apr 11, 2019

I’m afraid it’s not going to make 6.0. Sorry.

@DmitryTsepelev DmitryTsepelev force-pushed the configure-store-per-attachment branch from e3471c0 to e7f798c Compare Oct 1, 2019
@gmcgibbon gmcgibbon merged commit 84d236d into rails:master Oct 1, 2019
2 checks passed
@Hareramrai
Copy link

Hareramrai commented Mar 12, 2020

Hi @DmitryTsepelev, From which version of rails, we can use this?

@DmitryTsepelev
Copy link
Contributor Author

DmitryTsepelev commented Mar 12, 2020

Hi @Hareramrai, it's still in the main CHANGELOG so I guess it will be released as part of 6.1.

@dnalbach
Copy link

dnalbach commented Apr 28, 2020

@DmitryTsepelev does your merged PR allow for different S3 bucket names to be usable per attachment, or just the toggle of storage types? Is a service "s3" or is it granular down to the bucket name?

@DmitryTsepelev
Copy link
Contributor Author

DmitryTsepelev commented Apr 28, 2020

@dnalbach The line has_one_attached :avatar, service: :s3 forces ActiveStorage to use the service called s3 from storage.yml, so you can use any options (bucket, region, etc.) that are available for the specified adapter. For instance, if you have two buckets avatars and cover_photos you can do the following:

# storage.yml
s3_avatars:
  service: S3
  bucket: "photos"
  access_key_id: "..."
  secret_access_key: "..."

s3_cover_photos:
  service: S3
  bucket: "cover_photos"
  access_key_id: "..."
  secret_access_key: "..."
class User < ApplicationRecord
  has_one_attached :avatar, service: :s3_avatars
  has_one_attached :cover_photo, service: :s3_cover_photos
end

@dnalbach
Copy link

dnalbach commented Apr 29, 2020

Thats awsome @DmitryTsepelev , exactly what I was hoping for. This is a BigDeal in the cloud world I live in. Desperately needed. Look forward to seeing your contribution released!

@jules2689
Copy link
Contributor

jules2689 commented Aug 24, 2020

Hi! This is a great feature and I really appreciate the work done on it! I've opted to start using master on my app to take advantage of this new feature.

One thing I did notice is that the ergonomics of the system becomes a bit awkward with testing environments. Originally I had:

class Model < ApplicationModel
 has_one_attached :thing, service: :s3
end

But my tests started trying to upload to S3 😅 It was a simple fix for now:

class Model < ApplicationModel
 has_one_attached :thing, service: (Rails.env.test? ? :local : :s3)
end

But I was wondering if you had thought of some better ergonomics for this given that a ternary statement for every override could become quite awkward. Other things I've considered are:

  1. Override the :s3 storage config in the YAML file with if/else ERB blocks (This feels awkward too)
  2. Service class that sets constants based on environment (e.g. Service::THING_SERVICE could point to :s3 in dev/prod and :local in test). This seems ok, but it's not a pattern that I see a ton in rails

Again, great feature - would love to hear if anyone had any thoughts on a better solution here 🙇

@DmitryTsepelev
Copy link
Contributor Author

DmitryTsepelev commented Aug 24, 2020

Hi @jules2689, I'm going to recommend you a trick that will work for your specific case only (s3/disk pair), but it will probably will save a lot of time. Have you heard about minio? It's an S3–compatible storage, you can run it locally and keep using "S3" both in prod and dev

@palkan
Copy link
Contributor

palkan commented Aug 24, 2020

Override the :s3 storage config in the YAML file with if/else ERB blocks (This feels awkward too)

A better option would be to support per-environment configuration in storage.yml and move from service-oriented to context-oriented semantics, e.g.:

# storage.yml
production:
  images:
    service: s3
  documents:
    service: gcloud

test:
  images:
    service: disk
  documents:
    service: disk

And then you just have:

# Note that name "storage" would match better than "service" here
has_one_attached :avatar, storage: :images

IMO, that's the Rails Way of dealing with this problem.

@swissonelabs
Copy link

swissonelabs commented Oct 12, 2020

I am upgrading an existing app to 6.1.0.alpha

Why doesn't rails active_storage:install rails db:migrate update my tables?

because of that I don't have service_name in my storage_blob and facing errors.

@swissonelabs
Copy link

swissonelabs commented Oct 12, 2020

rails active_storage:update
worked!

@ghiculescu
Copy link
Member

ghiculescu commented Dec 10, 2020

rails active_storage:update
worked!

Should this be in the upgrade guide? See https://discuss.rubyonrails.org/t/problem-with-activestorage-blob-on-upgrade-from-6-0-3-4-to-6-1

@gokninski
Copy link

gokninski commented Jun 23, 2021

Hi! This is a great feature and I really appreciate the work done on it! I've opted to start using master on my app to take advantage of this new feature.

One thing I did notice is that the ergonomics of the system becomes a bit awkward with testing environments. Originally I had:

class Model < ApplicationModel
 has_one_attached :thing, service: :s3
end

But my tests started trying to upload to S3 😅 It was a simple fix for now:

class Model < ApplicationModel
 has_one_attached :thing, service: (Rails.env.test? ? :local : :s3)
end

But I was wondering if you had thought of some better ergonomics for this given that a ternary statement for every override could become quite awkward. Other things I've considered are:

  1. Override the :s3 storage config in the YAML file with if/else ERB blocks (This feels awkward too)
  2. Service class that sets constants based on environment (e.g. Service::THING_SERVICE could point to :s3 in dev/prod and :local in test). This seems ok, but it's not a pattern that I see a ton in rails

Again, great feature - would love to hear if anyone had any thoughts on a better solution here 🙇

I've done this using custom configuration in the environment config file (https://guides.rubyonrails.org/configuring.html#custom-configuration)

jyoun-godaddy pushed a commit to jyoun-godaddy/activestorage that referenced this pull request Jul 5, 2022
…igration

In rails/rails#33419, we added this migration to
properly upgrade Active Storage from 5.2 to 6.0

On Rails 6.1 `rails app:update` shouldn't add this migration to users' app.

Note that, I've left implementation that makes `rails app:update` to generate
migrations for users' app that are in `activestorage/db/update_migrate/`
because we are likely to need it e.g.: rails/rails#34935, rails/rails#36835.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet