Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ActiveStorage Guide #31037

Merged
merged 29 commits into from Dec 15, 2017
Merged

Conversation

jeffreyguenther
Copy link
Contributor

@jeffreyguenther jeffreyguenther commented Nov 3, 2017

Summary

This PR is meant to be a placeholder for the ActiveStorage guide.

Having implemented a service for OpenStack, I have a sense of what's necessary to build a new service and to integrate ActiveStorage into an app.

This PR is a work in progress and I hope to gather feedback as the guide comes together.

Topics:

  • How to attach a file(s) to a model.
  • How to remove the attached file.
  • How to link to the attached file.
  • How to implement a download link.
  • How to create variations of an image.
  • How to generate a preview for files other than images.
  • How to upload files directly to a service.
  • How to add support for additional cloud services.
  • How to clean up files during testing - use lessons learned in Rack app error when using SystemTestCase alongside ActiveStorage #30935
  • Example how to use direct upload events in a form

@rails-bot
Copy link

Thanks for the pull request, and welcome! The Rails team is excited to review your changes, and you should hear from @pixeltrix (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

This repository is being automatically checked for code quality issues using Code Climate. You can see results for this analysis in the PR status below. Newly introduced issues should be fixed before a Pull Request is considered ready to review.

Please see the contribution instructions for more information.

@rafaelfranca rafaelfranca added this to the 5.2.0 milestone Nov 6, 2017
@rafaelfranca
Copy link
Member

r? @georgeclaghorn

@georgeclaghorn
Copy link
Contributor

Thanks for kicking this off, @jeffreyguenther! ❤️ I’ll give a thorough review before the end of the week.

Copy link
Contributor

@georgeclaghorn georgeclaghorn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here’s a first pass. I have more to add later. Great start!

Active Storage
==============

This guide covers how to attach files to your ActiveRecord models.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use “Active Record” (with a space between the two words) to refer to the framework.

* How to create variations of an image.
* How to generate a preview for files other than images.
* How to upload files directly to a service.
* How to implement a download link.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This item is redundant. See “How to link to an attached file” above.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you thinking that "How to link to an attached file" should include a description how to create a download link with an attachment disposition?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think we can explain the disposition: option when we show how to generate a URL.

* How to link to the attached file.
* How to create variations of an image.
* How to generate a preview for files other than images.
* How to upload files directly to a service.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

* How to attach one or many files to a record.
* How to delete an attached file.
* How to link to an attached file.
* How to use variants to transform images.
* How to generate an image representation of a non-image file, such as a PDF or a video.
* How to send file uploads directly from browsers to a storage service, bypassing your application servers.

* How to generate a preview for files other than images.
* How to upload files directly to a service.
* How to implement a download link.
* How to add support for additional cloud services.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

* How to implement support for additional storage services.

You can associate the same blob with multiple application models as well. And if
you want to do transformations of a given `Blob`, the idea is that you'll simply
create a new one, rather than attempt to mutate the existing one (though of
course you can delete the previous version later if you don't need it).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let’s leave this entire section out.


Image files can furthermore be transformed using on-demand variants for quality,
aspect ratio, size, or any other
[MiniMagick](https://github.com/minimagick/minimagick) supported transformation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is Active Storage?
-----------------------

Active Storage facilitates uploading files to a cloud storage service like Amazon S3, Google Cloud
Storage, or Microsoft Azure Storage and attaching those files to Active Record objects. It comes
with a local disk-based service for development and testing and supports mirroring files to
subordinate services for backups and migrations.

Using Active Storage, an application can transform image uploads with
[ImageMagick](https://www.imagemagick.org), generate image representations of non-image uploads
like PDFs and videos, and extract metadata from arbitrary files.

root: <%= Rails.root.join("storage") %>
```
NOTE: Should we include the required keys for all the supported services?
NOTE: Should we mention the mirror service and how to set it up?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes and yes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To expand on my answer to the first question, let’s show the minimum required config for each service and refer readers to client library docs for optional config.

-------------------------------

To remove an attachment from a model, call `purge` on the attachment. Removal
can be done in the background if your application is setup to use ActiveJob.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setup is a noun. The verb form is “set up.”

Use “Active Job” (with a space between the words) to refer to the framework.


# Destroy the associated models and actual resource files async, via Active Job.
user.avatar.purge_later
```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let’s be clear that purging an attachment deletes its underlying blob and removes the stored file from the service.

access, a redirect to the actual service endpoint is returned. This indirection
decouples the public URL from the actual one, and allows for example mirroring
attachments in different services for high-availability. The redirection has an
HTTP expiration of 5 min.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This indirection decouples the public URL from the actual one, and allows for example mirroring attachments in different services for high-availability.

The primary reason for pointing to a stable application URL is to support page and fragment caching. A short-lived URL like one returned by ActiveStorage::Service#service_url can’t be cached. I’d just say that: “decoupling” is too abstract a benefit for a usage guide, and mirroring isn’t (as far as I can tell) a good justification for the redirect.

@jeffreyguenther
Copy link
Contributor Author

Thanks @georgeclaghorn! I'll work on these!

- Describes how to setup each of the services in the `services.yml`
- Integrates copy changes
@jeffreyguenther
Copy link
Contributor Author

I haven't addressed the download link yet as I'm a little fuzzy on how to do this. url_for, as far as I know, doesn't allow you to pass it params when you're passing it an object. Is that correct?

When I recently implemented a download link, I created a route + controller to handle the download and it passed the disposition into service_url. Have I missed how this should be done?

bucket: ""
```

Also, add the S3 client gem to your Gemfile:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/S3/Google Cloud Storage/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! Copy and paste error.

@georgeclaghorn
Copy link
Contributor

@jeffreyguenther That sounds correct. We can recommend rails_blob_url when params need to be provided.

@pacMakaveli
Copy link

Hi guys.

Considering most of the Rails community has been using Paperclip ( mainly ) to add attachments in their app, will there ever be a migration guide from Paperclip to ActiveStorage? Or maybe, even better imo, does ActiveJob provide an option to customize the S3 path?
Is this even something you guys would cover!?

Thanks!

@jeffreyguenther
Copy link
Contributor Author

@pacMakaveli I maintain a couple apps that use Paperclip myself so I have given some thought to the problem. I don’t have any plans to write anything to that end. In my view, it’s outside the scope of this guide.

Paperclip and Active Storage have quite different approaches to the way files are organized on the cloud service, so the process will be involved, even more so if you try to preserve your formats as variants. Personally, I wouldn’t do that. I would only move the originals and let Active Storage generate the variants on demand as your application is used.

@jeffreyguenther
Copy link
Contributor Author

@georgeclaghorn Should it be possible for users to set the filename of the file downloaded?

BlobsController currently only passes the disposition to the service_url and uses the Blob's file name as the download file name. Service#url does take a filename so it is possible with a small change.

```
3. That's it! Uploads begin upon form submission.

### Direct upload JavaScript events
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's an example of how you might use these events in an app to display direct upload progress: https://gist.github.com/javan/5538692cb37a683db15792be8d05761e. Feel free to borrow (or ignore)!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Good idea! I'll incorporate your gist. I find the more tangible examples we can provide the better people will understand what's possible.

@georgeclaghorn
Copy link
Contributor

Fixed the route helper bug in 3fa8126.

@tradeli
Copy link

tradeli commented Nov 28, 2017

Per this draft, it looks like ActiveStorage is only useful with non-API Rails apps because of the Active View integration, right?

@rafaelfranca
Copy link
Member

rafaelfranca commented Nov 28, 2017

@tradeli no. Of course you will not take fully advantage of the Active Strorage features, but you can use it without Action View.

To setup an existing application after upgrading to Rails 5.2, run `rails
active_storage:install`. If you're creating a new project with Rails 5.2,
ActiveStorage will be installed by default. Installation generates a migration
to add the tables needed to store attachments.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarify the kind of installation required:

Active Storage uses two tables in your application’s database named active_storage_blobs and active_storage_attachments. After upgrading your application to Rails 5.2, run rails active_storage:install to generate a migration that creates these tables. Use rails db:migrate to run the migration.

You need not run rails active_storage:install in a new Rails 5.2 application: the migration is generated automatically.


Inside a Rails application, you can set up your services through the generated
`config/storage.yml` file and reference one of the supported service types under
the `service` key.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Elaborate on the structure of config/storage.yml and provide an example:

Declare Active Storage services in config/storage.yml. For each service your application uses, provide a name and the requisite configuration. The example below declares three services named local, test, and s3:

local:
  service: Disk
  root: <%= Rails.root.join("storage") %>

test:
  service: Disk
  root: <%= Rails.root.join("tmp/storage") %>

s3:
  service: S3
  access_key_id: ""
  secret_access_key: ""

Continue reading for information on the built-in service adapters (e.g. Disk and S3) and the configuration they require.

Expand on how and when to specify which service Active Storage should use:

Tell Active Storage which service to use by setting Rails.application.config.active_storage.service. Because each environment will likely use a different service, it is recommended to do this on a per-environment basis. To use the disk service from the previous example in the development environment, you would add the following to config/environments/development.rb:

# Store files locally.
config.active_storage.service = :disk

To use the s3 service in production, you would add the following to config/environments/production.rb:

# Store files in S3.
config.active_storage.service = :s3

Because populating config/storage.yml and setting Rails.application.config.active_storage.service are required and using image variants is optional, I would put the information on adding mini_magick to the Gemfile last.

the `service` key.

### Disk Service
To use the Disk service:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let’s clarify that the Disk service is intended for local use only. It shouldn’t be used in production.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about for small apps that don't warrant setting up cloud storage? Other gems like Paperclip allow local storage. Are there performance reasons why this shouldn't be done?


To use Amazon S3:
``` yaml
local:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Give this a different name. local isn’t quite right.

To use Microsoft Azure Storage:

``` yaml
local:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Give this a different name. local isn’t quite right.

To use Google Cloud Storage:

``` yaml
local:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Give this a different name. local isn’t quite right.

You can keep multiple services in sync by defining a mirror service. When
a file is uploaded or deleted, it's done across all the mirrored services.
Define each of the services you'd like to use as described above and then define
a mirrored service which references them.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Explain that files are always served from the primary service. Only writes are mirrored.

Comment on why you’d use the Mirror service. It’s useful in carrying out a migration between services in production. You can start mirroring to the new service, copy existing files from the old service to the new, then go all-in on the new service.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't

def delete(key)
perform_across_services :delete, key
end

mean that deletes are done across all services?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A delete is a write. 😄

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Touché! 😁

@jeffreyguenther
Copy link
Contributor Author

With DHH's announcement of the 5.2 beta, it looks like we're getting close to a release. I'll do my best to get these edits done in the next day. Hate to be blocking a release!


Some services are supported by community maintained gems:

* [OpenStack](https://github.com/jeffreyguenther/activestorage-openstack)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have other lists like this in the Rails guides? I worry that it adds an unnecessary review burden going forward unless our policy is to add all submitted 3rd party libs.

I appreciate that you created this gem though! ❤️

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe there are similar lists in Active Job.

https://github.com/rails/rails/blob/master/guides/source/active_job_basics.md#L163-L167

This is why I felt comfortable adding it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm game to remove the section if the powers that be say that's what we should. If guides are only updated on a per release basis, I think that's probably best.

@jeffreyguenther
Copy link
Contributor Author

Ready for final review.

development environment, you would add the following to
config/environments/development.rb:

In your application's configuration, specify the service to use like this:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✂️ this line.


``` ruby
gem 'mini_magick'
```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, when I said to put this last, I meant before the service information (last in the bit of text under the “Setup“ heading).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ya, I followed that, but it didn't feel right to me.

We say:

Continue reading for more information on the built-in service adapters (e.g.
Disk and S3) and the configuration they require.

As a reader, this led me to think that I would be receiving info about the different service types right away. It flowed better, to me, to have it below the mirrored service description.

I'll put it back the way you had it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see what you mean. Maybe we should move this information into the section on transforming images.

Add Support Additional Cloud Service
------------------------------------

ActiveStorage ships with support for Amazon S3, Google Cloud Storage, and Azure.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let’s not repeat this information here. I’d just start the next sentence like this: “If you need to support a cloud service other than those with built-in support, …“

`Disk` and `S3`) and the configuration they require.

### Disk Service
To use the Disk service:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

“Declare a Disk service in config/storage.yml:”


### Amazon S3 Service

To use Amazon S3:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

“Declare an S3 service in config/storage.yml:”

```
### Microsoft Azure Storage Service

To use Microsoft Azure Storage:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

“Declare an Azure Storage service in config/storage.yml:”


### Google Cloud Storage Service

To use Google Cloud Storage:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

“Declare a Google Cloud Storage service in config/storage.yml:”

params.require(:message).permit(:title, :content, images: [])
end
end
```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can afford to expand on this, given that it’s the central feature of Active Storage:

Attach files to records

has_one_attached

The has_one_attached macro sets up a one-to-one mapping between records and files. Each record can have one file attached to it.

For example, suppose your application has a User model. If you want each user to have an avatar, define the User model like this:

class User < ApplicationRecord
  has_one_attached :avatar
end

You can create a user with an avatar:

class SignupController < ApplicationController
  def create
    user = Users.create!(user_params)
    session[:user_id] = user.id
    redirect_to root_path
  end

  private
    def user_params
      params.require(:user).permit(:email_address, :password, :avatar)
    end
end

Call avatar.attach to attach an avatar to an existing user:

Current.user.avatar.attach(params[:avatar])

Call avatar.attached? to determine whether a particular user has an avatar:

Current.user.avatar.attached?

has_many_attached

The has_many_attached macro sets up a one-to-many relationship between records and files. Each record can have many files attached to it.

For example, suppose your application has a Message model. If you want each message to have many images, define the Message model like this:

class Message < ApplicationRecord
  has_many_attached :images
end

You can create a message with images:

class MessagesController < ApplicationController
  def create
    message = Message.create!(message_params)
    redirect_to message
  end

  private
    def message_params
      params.require(:message).permit(:title, :content, images: [])
    end
end

Call images.attach to add new images to an existing message:

@message.images.attach(params[:images])

Call images.attached? to determine whether a particular message has any images:

@message.images.attached?

config.active_storage.service = :local_test
```

Add Support Additional Cloud Service

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a native English speaker, but this doesn't sound right. Maybe Add Support for Additional Cloud Services or Add Support for an Additional Cloud Service

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. That's a typo!

Add Support Additional Cloud Service
------------------------------------

If you need to support a cloud service other these, you will need to implement

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.. a cloud service other than these, you ..

@message.images.attach(params[:images])
```

Call `images.attached?`` to determine whether a particular message has any images:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extra `

backups and migrations.

Using Active Storage, an application can transform image uploads with
[ImageMagick](https://www.imagemagick.org), generate image representations of

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which one is it? ImageMagick or MiniMagic or both? On line 329 you mention MiniMagick for creating variants.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MiniMagick is a ruby library wrapping the commandline utility ImageMagick. The args you pass to MiniMagick are the same as what you use on the commandline and the docs live over at ImageMagick explaining what they do. You're using ImageMagick via the MiniMagick.

end
```

If your system tests verify the deletion of a model with attachments and your

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... with attachments and you're using ...

Copy link
Contributor

@georgeclaghorn georgeclaghorn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is my last round of review. Thanks so much for your work on this!

`Disk` and `S3`) and the configuration they require.

### Disk Service
Declare a Disk service in `config/storage.yml`:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leave an extra line between the heading above and this paragraph.

root: <%= Rails.root.join("storage") %>
```

### Amazon S3 Service
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leave an extra line under this heading.

``` ruby
gem "aws-sdk-s3", require: false
```
### Microsoft Azure Storage Service
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.