New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ActiveStorage Guide #31037

Merged
merged 29 commits into from Dec 15, 2017

Conversation

@jeffreyguenther
Contributor

jeffreyguenther commented Nov 3, 2017

Summary

This PR is meant to be a placeholder for the ActiveStorage guide.

Having implemented a service for OpenStack, I have a sense of what's necessary to build a new service and to integrate ActiveStorage into an app.

This PR is a work in progress and I hope to gather feedback as the guide comes together.

Topics:

  • How to attach a file(s) to a model.
  • How to remove the attached file.
  • How to link to the attached file.
  • How to implement a download link.
  • How to create variations of an image.
  • How to generate a preview for files other than images.
  • How to upload files directly to a service.
  • How to add support for additional cloud services.
  • How to clean up files during testing - use lessons learned in #30935
  • Example how to use direct upload events in a form
@rails-bot

This comment has been minimized.

rails-bot commented Nov 3, 2017

Thanks for the pull request, and welcome! The Rails team is excited to review your changes, and you should hear from @pixeltrix (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

This repository is being automatically checked for code quality issues using Code Climate. You can see results for this analysis in the PR status below. Newly introduced issues should be fixed before a Pull Request is considered ready to review.

Please see the contribution instructions for more information.

@rafaelfranca rafaelfranca added this to the 5.2.0 milestone Nov 6, 2017

@rafaelfranca

This comment has been minimized.

Member

rafaelfranca commented Nov 9, 2017

@rails-bot rails-bot assigned georgeclaghorn and unassigned pixeltrix Nov 9, 2017

@georgeclaghorn

This comment has been minimized.

Member

georgeclaghorn commented Nov 14, 2017

Thanks for kicking this off, @jeffreyguenther! ❤️ I’ll give a thorough review before the end of the week.

@georgeclaghorn

Here’s a first pass. I have more to add later. Great start!

Active Storage
==============
This guide covers how to attach files to your ActiveRecord models.

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Nov 17, 2017

Member

Use “Active Record” (with a space between the two words) to refer to the framework.

* How to create variations of an image.
* How to generate a preview for files other than images.
* How to upload files directly to a service.
* How to implement a download link.

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Nov 17, 2017

Member

This item is redundant. See “How to link to an attached file” above.

This comment has been minimized.

@jeffreyguenther

jeffreyguenther Nov 17, 2017

Contributor

Are you thinking that "How to link to an attached file" should include a description how to create a download link with an attachment disposition?

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Nov 17, 2017

Member

Yeah, I think we can explain the disposition: option when we show how to generate a URL.

* How to link to the attached file.
* How to create variations of an image.
* How to generate a preview for files other than images.
* How to upload files directly to a service.

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Nov 17, 2017

Member
* How to attach one or many files to a record.
* How to delete an attached file.
* How to link to an attached file.
* How to use variants to transform images.
* How to generate an image representation of a non-image file, such as a PDF or a video.
* How to send file uploads directly from browsers to a storage service, bypassing your application servers.
* How to generate a preview for files other than images.
* How to upload files directly to a service.
* How to implement a download link.
* How to add support for additional cloud services.

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Nov 17, 2017

Member
* How to implement support for additional storage services.
You can associate the same blob with multiple application models as well. And if
you want to do transformations of a given `Blob`, the idea is that you'll simply
create a new one, rather than attempt to mutate the existing one (though of
course you can delete the previous version later if you don't need it).

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Nov 17, 2017

Member

Let’s leave this entire section out.

Image files can furthermore be transformed using on-demand variants for quality,
aspect ratio, size, or any other
[MiniMagick](https://github.com/minimagick/minimagick) supported transformation.

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Nov 17, 2017

Member
What is Active Storage?
-----------------------

Active Storage facilitates uploading files to a cloud storage service like Amazon S3, Google Cloud
Storage, or Microsoft Azure Storage and attaching those files to Active Record objects. It comes
with a local disk-based service for development and testing and supports mirroring files to
subordinate services for backups and migrations.

Using Active Storage, an application can transform image uploads with
[ImageMagick](https://www.imagemagick.org), generate image representations of non-image uploads
like PDFs and videos, and extract metadata from arbitrary files.
root: <%= Rails.root.join("storage") %>
```
NOTE: Should we include the required keys for all the supported services?
NOTE: Should we mention the mirror service and how to set it up?

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Nov 17, 2017

Member

Yes and yes.

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Nov 17, 2017

Member

To expand on my answer to the first question, let’s show the minimum required config for each service and refer readers to client library docs for optional config.

-------------------------------
To remove an attachment from a model, call `purge` on the attachment. Removal
can be done in the background if your application is setup to use ActiveJob.

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Nov 17, 2017

Member

Setup is a noun. The verb form is “set up.”

Use “Active Job” (with a space between the words) to refer to the framework.

# Destroy the associated models and actual resource files async, via Active Job.
user.avatar.purge_later
```

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Nov 17, 2017

Member

Let’s be clear that purging an attachment deletes its underlying blob and removes the stored file from the service.

access, a redirect to the actual service endpoint is returned. This indirection
decouples the public URL from the actual one, and allows for example mirroring
attachments in different services for high-availability. The redirection has an
HTTP expiration of 5 min.

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Nov 17, 2017

Member

This indirection decouples the public URL from the actual one, and allows for example mirroring attachments in different services for high-availability.

The primary reason for pointing to a stable application URL is to support page and fragment caching. A short-lived URL like one returned by ActiveStorage::Service#service_url can’t be cached. I’d just say that: “decoupling” is too abstract a benefit for a usage guide, and mirroring isn’t (as far as I can tell) a good justification for the redirect.

@jeffreyguenther

This comment has been minimized.

Contributor

jeffreyguenther commented Nov 17, 2017

Thanks @georgeclaghorn! I'll work on these!

Integrates George's suggestions
- Describes how to setup each of the services in the `services.yml`
- Integrates copy changes
@jeffreyguenther

This comment has been minimized.

Contributor

jeffreyguenther commented Nov 19, 2017

I haven't addressed the download link yet as I'm a little fuzzy on how to do this. url_for, as far as I know, doesn't allow you to pass it params when you're passing it an object. Is that correct?

When I recently implemented a download link, I created a route + controller to handle the download and it passed the disposition into service_url. Have I missed how this should be done?

bucket: ""
```
Also, add the S3 client gem to your Gemfile:

This comment has been minimized.

@simonista

simonista Nov 19, 2017

s/S3/Google Cloud Storage/

This comment has been minimized.

@jeffreyguenther

jeffreyguenther Nov 19, 2017

Contributor

Good catch! Copy and paste error.

@georgeclaghorn

This comment has been minimized.

Member

georgeclaghorn commented Nov 19, 2017

@jeffreyguenther That sounds correct. We can recommend rails_blob_url when params need to be provided.

@pacMakaveli

This comment has been minimized.

pacMakaveli commented Nov 19, 2017

Hi guys.

Considering most of the Rails community has been using Paperclip ( mainly ) to add attachments in their app, will there ever be a migration guide from Paperclip to ActiveStorage? Or maybe, even better imo, does ActiveJob provide an option to customize the S3 path?
Is this even something you guys would cover!?

Thanks!

@jeffreyguenther

This comment has been minimized.

Contributor

jeffreyguenther commented Nov 19, 2017

@pacMakaveli I maintain a couple apps that use Paperclip myself so I have given some thought to the problem. I don’t have any plans to write anything to that end. In my view, it’s outside the scope of this guide.

Paperclip and Active Storage have quite different approaches to the way files are organized on the cloud service, so the process will be involved, even more so if you try to preserve your formats as variants. Personally, I wouldn’t do that. I would only move the originals and let Active Storage generate the variants on demand as your application is used.

@jeffreyguenther

This comment has been minimized.

Contributor

jeffreyguenther commented Nov 21, 2017

@georgeclaghorn Should it be possible for users to set the filename of the file downloaded?

BlobsController currently only passes the disposition to the service_url and uses the Blob's file name as the download file name. Service#url does take a filename so it is possible with a small change.

```
3. That's it! Uploads begin upon form submission.
### Direct upload JavaScript events

This comment has been minimized.

@javan

javan Nov 21, 2017

Member

Here's an example of how you might use these events in an app to display direct upload progress: https://gist.github.com/javan/5538692cb37a683db15792be8d05761e. Feel free to borrow (or ignore)!

This comment has been minimized.

@jeffreyguenther

jeffreyguenther Nov 21, 2017

Contributor

Thanks! Good idea! I'll incorporate your gist. I find the more tangible examples we can provide the better people will understand what's possible.

@georgeclaghorn

This comment has been minimized.

Member

georgeclaghorn commented Nov 21, 2017

Fixed the route helper bug in 3fa8126.

@tradeli

This comment has been minimized.

tradeli commented Nov 28, 2017

Per this draft, it looks like ActiveStorage is only useful with non-API Rails apps because of the Active View integration, right?

@rafaelfranca

This comment has been minimized.

Member

rafaelfranca commented Nov 28, 2017

@tradeli no. Of course you will not take fully advantage of the Active Strorage features, but you can use it without Action View.

To setup an existing application after upgrading to Rails 5.2, run `rails
active_storage:install`. If you're creating a new project with Rails 5.2,
ActiveStorage will be installed by default. Installation generates a migration
to add the tables needed to store attachments.

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Nov 29, 2017

Member

Clarify the kind of installation required:

Active Storage uses two tables in your application’s database named active_storage_blobs and active_storage_attachments. After upgrading your application to Rails 5.2, run rails active_storage:install to generate a migration that creates these tables. Use rails db:migrate to run the migration.

You need not run rails active_storage:install in a new Rails 5.2 application: the migration is generated automatically.

Inside a Rails application, you can set up your services through the generated
`config/storage.yml` file and reference one of the supported service types under
the `service` key.

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Nov 29, 2017

Member

Elaborate on the structure of config/storage.yml and provide an example:

Declare Active Storage services in config/storage.yml. For each service your application uses, provide a name and the requisite configuration. The example below declares three services named local, test, and s3:

local:
  service: Disk
  root: <%= Rails.root.join("storage") %>

test:
  service: Disk
  root: <%= Rails.root.join("tmp/storage") %>

s3:
  service: S3
  access_key_id: ""
  secret_access_key: ""

Continue reading for information on the built-in service adapters (e.g. Disk and S3) and the configuration they require.

Expand on how and when to specify which service Active Storage should use:

Tell Active Storage which service to use by setting Rails.application.config.active_storage.service. Because each environment will likely use a different service, it is recommended to do this on a per-environment basis. To use the disk service from the previous example in the development environment, you would add the following to config/environments/development.rb:

# Store files locally.
config.active_storage.service = :disk

To use the s3 service in production, you would add the following to config/environments/production.rb:

# Store files in S3.
config.active_storage.service = :s3

Because populating config/storage.yml and setting Rails.application.config.active_storage.service are required and using image variants is optional, I would put the information on adding mini_magick to the Gemfile last.

the `service` key.
### Disk Service
To use the Disk service:

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Nov 29, 2017

Member

Let’s clarify that the Disk service is intended for local use only. It shouldn’t be used in production.

This comment has been minimized.

@jeffreyguenther

jeffreyguenther Nov 30, 2017

Contributor

What about for small apps that don't warrant setting up cloud storage? Other gems like Paperclip allow local storage. Are there performance reasons why this shouldn't be done?

To use Amazon S3:
``` yaml
local:

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Nov 29, 2017

Member

Give this a different name. local isn’t quite right.

To use Microsoft Azure Storage:
``` yaml
local:

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Nov 29, 2017

Member

Give this a different name. local isn’t quite right.

To use Google Cloud Storage:
``` yaml
local:

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Nov 29, 2017

Member

Give this a different name. local isn’t quite right.

You can keep multiple services in sync by defining a mirror service. When
a file is uploaded or deleted, it's done across all the mirrored services.
Define each of the services you'd like to use as described above and then define
a mirrored service which references them.

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Nov 29, 2017

Member

Explain that files are always served from the primary service. Only writes are mirrored.

Comment on why you’d use the Mirror service. It’s useful in carrying out a migration between services in production. You can start mirroring to the new service, copy existing files from the old service to the new, then go all-in on the new service.

This comment has been minimized.

@jeffreyguenther

jeffreyguenther Nov 30, 2017

Contributor

Doesn't

def delete(key)
perform_across_services :delete, key
end

mean that deletes are done across all services?

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Nov 30, 2017

Member

A delete is a write. 😄

This comment has been minimized.

@jeffreyguenther

jeffreyguenther Nov 30, 2017

Contributor

Touché! 😁

@jeffreyguenther

This comment has been minimized.

Contributor

jeffreyguenther commented Nov 29, 2017

With DHH's announcement of the 5.2 beta, it looks like we're getting close to a release. I'll do my best to get these edits done in the next day. Hate to be blocking a release!

Some services are supported by community maintained gems:
* [OpenStack](https://github.com/jeffreyguenther/activestorage-openstack)

This comment has been minimized.

@javan

javan Nov 30, 2017

Member

Do we have other lists like this in the Rails guides? I worry that it adds an unnecessary review burden going forward unless our policy is to add all submitted 3rd party libs.

I appreciate that you created this gem though! ❤️

This comment has been minimized.

@jeffreyguenther

jeffreyguenther Nov 30, 2017

Contributor

I believe there are similar lists in Active Job.

https://github.com/rails/rails/blob/master/guides/source/active_job_basics.md#L163-L167

This is why I felt comfortable adding it.

This comment has been minimized.

@jeffreyguenther

jeffreyguenther Nov 30, 2017

Contributor

I'm game to remove the section if the powers that be say that's what we should. If guides are only updated on a per release basis, I think that's probably best.

@jeffreyguenther

This comment has been minimized.

Contributor

jeffreyguenther commented Nov 30, 2017

Ready for final review.

development environment, you would add the following to
config/environments/development.rb:
In your application's configuration, specify the service to use like this:

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Dec 4, 2017

Member

✂️ this line.

``` ruby
gem 'mini_magick'
```

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Dec 4, 2017

Member

Sorry, when I said to put this last, I meant before the service information (last in the bit of text under the “Setup“ heading).

This comment has been minimized.

@jeffreyguenther

jeffreyguenther Dec 5, 2017

Contributor

Ya, I followed that, but it didn't feel right to me.

We say:

Continue reading for more information on the built-in service adapters (e.g.
Disk and S3) and the configuration they require.

As a reader, this led me to think that I would be receiving info about the different service types right away. It flowed better, to me, to have it below the mirrored service description.

I'll put it back the way you had it.

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Dec 5, 2017

Member

I see what you mean. Maybe we should move this information into the section on transforming images.

Add Support Additional Cloud Service
------------------------------------
ActiveStorage ships with support for Amazon S3, Google Cloud Storage, and Azure.

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Dec 4, 2017

Member

Let’s not repeat this information here. I’d just start the next sentence like this: “If you need to support a cloud service other than those with built-in support, …“

`Disk` and `S3`) and the configuration they require.
### Disk Service
To use the Disk service:

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Dec 4, 2017

Member

“Declare a Disk service in config/storage.yml:”

### Amazon S3 Service
To use Amazon S3:

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Dec 4, 2017

Member

“Declare an S3 service in config/storage.yml:”

```
### Microsoft Azure Storage Service
To use Microsoft Azure Storage:

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Dec 4, 2017

Member

“Declare an Azure Storage service in config/storage.yml:”

### Google Cloud Storage Service
To use Google Cloud Storage:

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Dec 4, 2017

Member

“Declare a Google Cloud Storage service in config/storage.yml:”

params.require(:message).permit(:title, :content, images: [])
end
end
```

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Dec 4, 2017

Member

We can afford to expand on this, given that it’s the central feature of Active Storage:

Attach files to records

has_one_attached

The has_one_attached macro sets up a one-to-one mapping between records and files. Each record can have one file attached to it.

For example, suppose your application has a User model. If you want each user to have an avatar, define the User model like this:

class User < ApplicationRecord
  has_one_attached :avatar
end

You can create a user with an avatar:

class SignupController < ApplicationController
  def create
    user = Users.create!(user_params)
    session[:user_id] = user.id
    redirect_to root_path
  end

  private
    def user_params
      params.require(:user).permit(:email_address, :password, :avatar)
    end
end

Call avatar.attach to attach an avatar to an existing user:

Current.user.avatar.attach(params[:avatar])

Call avatar.attached? to determine whether a particular user has an avatar:

Current.user.avatar.attached?

has_many_attached

The has_many_attached macro sets up a one-to-many relationship between records and files. Each record can have many files attached to it.

For example, suppose your application has a Message model. If you want each message to have many images, define the Message model like this:

class Message < ApplicationRecord
  has_many_attached :images
end

You can create a message with images:

class MessagesController < ApplicationController
  def create
    message = Message.create!(message_params)
    redirect_to message
  end

  private
    def message_params
      params.require(:message).permit(:title, :content, images: [])
    end
end

Call images.attach to add new images to an existing message:

@message.images.attach(params[:images])

Call images.attached? to determine whether a particular message has any images:

@message.images.attached?
config.active_storage.service = :local_test
```
Add Support Additional Cloud Service

This comment has been minimized.

@pacMakaveli

pacMakaveli Dec 6, 2017

I'm not a native English speaker, but this doesn't sound right. Maybe Add Support for Additional Cloud Services or Add Support for an Additional Cloud Service

This comment has been minimized.

@jeffreyguenther

jeffreyguenther Dec 6, 2017

Contributor

Good catch. That's a typo!

Add Support Additional Cloud Service
------------------------------------
If you need to support a cloud service other these, you will need to implement

This comment has been minimized.

@pacMakaveli

pacMakaveli Dec 6, 2017

.. a cloud service other than these, you ..

@message.images.attach(params[:images])
```
Call `images.attached?`` to determine whether a particular message has any images:

This comment has been minimized.

@pacMakaveli
backups and migrations.
Using Active Storage, an application can transform image uploads with
[ImageMagick](https://www.imagemagick.org), generate image representations of

This comment has been minimized.

@pacMakaveli

pacMakaveli Dec 6, 2017

Which one is it? ImageMagick or MiniMagic or both? On line 329 you mention MiniMagick for creating variants.

This comment has been minimized.

@jeffreyguenther

jeffreyguenther Dec 6, 2017

Contributor

MiniMagick is a ruby library wrapping the commandline utility ImageMagick. The args you pass to MiniMagick are the same as what you use on the commandline and the docs live over at ImageMagick explaining what they do. You're using ImageMagick via the MiniMagick.

end
```
If your system tests verify the deletion of a model with attachments and your

This comment has been minimized.

@pacMakaveli

pacMakaveli Dec 6, 2017

... with attachments and you're using ...

@georgeclaghorn

This is my last round of review. Thanks so much for your work on this!

`Disk` and `S3`) and the configuration they require.
### Disk Service
Declare a Disk service in `config/storage.yml`:

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Dec 14, 2017

Member

Leave an extra line between the heading above and this paragraph.

root: <%= Rails.root.join("storage") %>
```
### Amazon S3 Service

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Dec 14, 2017

Member

Leave an extra line under this heading.

``` ruby
gem "aws-sdk-s3", require: false
```
### Microsoft Azure Storage Service

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Dec 14, 2017

Member

Leave an extra line under this heading.

gem "azure-storage", require: false
```
### Google Cloud Storage Service

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Dec 14, 2017

Member

Leave an extra line under this heading.

NOTE: Files are served from the primary service.
Attach Files to a Model
--------------------------

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Dec 14, 2017

Member

Match the number of dashes to the length of the heading.

-----------------------------------
Sometimes your application requires images in a different format than
what was uploaded. To create variation of the image, call `variant` on the Blob.

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Dec 14, 2017

Member

To create a variation of an image…

<%= image_tag user.avatar.variant(resize: "100x100") %>
```
Create Image Previews of Attachments

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Dec 14, 2017

Member

Preview Non-Image Files

Create Image Previews of Attachments
------------------------------------
Previews can be generated from some non-image formats. ActiveStorage supports
Previewers for videos and PDFs.

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Dec 14, 2017

Member

Leave an extra line between the heading above and this paragraph.

Previewers are an implementation detail that I’d prefer not to expose in a guide intended for newcomers:

Some non-image files can be previewed: that is, they can be presented as images. For example, a video file can be previewed by extracting its first frame. Out of the box, Active Storage supports previewing videos and PDF documents.

config.active_storage.service = :local_test
```
Add Support for Additional Cloud Services

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Dec 14, 2017

Member

Support Additional Storage Services

```
Add Support for Additional Cloud Services
------------------------------------

This comment has been minimized.

@georgeclaghorn

georgeclaghorn Dec 14, 2017

Member

Match the number of dashes to the length of the heading.

@jeffreyguenther

This comment has been minimized.

Contributor

jeffreyguenther commented Dec 15, 2017

Here you go!

@georgeclaghorn georgeclaghorn merged commit 48fbc4a into rails:master Dec 15, 2017

1 of 2 checks passed

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
codeclimate All good!
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment