Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add live uploads #1184

Merged
merged 157 commits into from
Nov 5, 2020
Merged
Show file tree
Hide file tree
Changes from 132 commits
Commits
Show all changes
157 commits
Select commit Hold shift + click to select a range
7011a3b
WIP: upload channel
Gazler Mar 28, 2019
a1bb53d
fix up after rebase
mcrumm Apr 20, 2020
ae21270
small js fixes
mcrumm Apr 24, 2020
703a94a
fix lv channel monitoring uploads
mcrumm Apr 24, 2020
9d0e97d
ensure upload events can target components
mcrumm May 10, 2020
0ad4593
skip patching file inputs
mcrumm May 10, 2020
8542bbe
WIP: prep for hooks
mcrumm Jun 8, 2020
cd61fce
WIP: bind file progress
mcrumm Jun 19, 2020
4742c3c
Uplaod config/entry start
chrismccord Jul 10, 2020
27f685b
wip: refactor preflight
mcrumm Jul 10, 2020
384f34e
WIP
chrismccord Jul 13, 2020
c375083
Refactor
chrismccord Jul 13, 2020
0288cff
Drop comments
chrismccord Jul 13, 2020
23bb1b3
Invoke onComplete form submit when uploads complete
chrismccord Jul 13, 2020
9bb047d
Comment
chrismccord Jul 13, 2020
4ac2bf1
Comment
chrismccord Jul 13, 2020
ed487b9
Comment
chrismccord Jul 13, 2020
537899c
Prep validations
chrismccord Jul 14, 2020
185b2c9
Stub casting an dvalidation of entries
chrismccord Jul 14, 2020
399c8bd
Validate max_entries in config
mcrumm Jul 14, 2020
1162471
Initial accept filter option
mcrumm Jul 14, 2020
3a3e447
more accept validation
mcrumm Jul 15, 2020
9d2126f
Validate entries by type or extension
mcrumm Jul 16, 2020
e9af4ca
Validate max_file_size per entry
mcrumm Jul 16, 2020
8f3eaf5
todo
mcrumm Jul 16, 2020
9271046
Log allow_upload error on client
mcrumm Jul 17, 2020
fb571d1
Put errors on config
mcrumm Jul 17, 2020
f972211
WIP: upload channel
Gazler Mar 28, 2019
483e90e
fix up after rebase
mcrumm Apr 20, 2020
d6b7826
small js fixes
mcrumm Apr 24, 2020
a8ff9f0
fix lv channel monitoring uploads
mcrumm Apr 24, 2020
4da9369
ensure upload events can target components
mcrumm May 10, 2020
4c4edf5
skip patching file inputs
mcrumm May 10, 2020
eed2272
WIP: prep for hooks
mcrumm Jun 8, 2020
c373516
WIP: bind file progress
mcrumm Jun 19, 2020
6816edb
Uplaod config/entry start
chrismccord Jul 10, 2020
13e5947
wip: refactor preflight
mcrumm Jul 10, 2020
3354f06
WIP
chrismccord Jul 13, 2020
8dd13ef
Refactor
chrismccord Jul 13, 2020
a70ca9d
Drop comments
chrismccord Jul 13, 2020
1f8d7ba
Invoke onComplete form submit when uploads complete
chrismccord Jul 13, 2020
e8e9031
Comment
chrismccord Jul 13, 2020
e61a705
Comment
chrismccord Jul 13, 2020
632ef21
Comment
chrismccord Jul 13, 2020
49187e6
Prep validations
chrismccord Jul 14, 2020
1b45d36
Stub casting an dvalidation of entries
chrismccord Jul 14, 2020
0221bff
Validate max_entries in config
mcrumm Jul 14, 2020
cba5a12
Initial accept filter option
mcrumm Jul 14, 2020
9a75138
more accept validation
mcrumm Jul 15, 2020
5fdf7b5
Validate entries by type or extension
mcrumm Jul 16, 2020
4d00acc
Validate max_file_size per entry
mcrumm Jul 16, 2020
9427081
todo
mcrumm Jul 16, 2020
205e549
Log allow_upload error on client
mcrumm Jul 17, 2020
b1a74b1
Put errors on config
mcrumm Jul 17, 2020
68321e2
Merge branch 'mc-uploads-next' of github.com:phoenixframework/phoenix…
chrismccord Jul 23, 2020
f4f2f6d
Add upload channel registration and initial integration tests
chrismccord Jul 27, 2020
bd4b747
Store entry refs to channel pids map on UploadConfig
chrismccord Aug 4, 2020
f50b56a
Add file_input to LiveViewTest
chrismccord Aug 5, 2020
87359eb
WIP working LiveViewTest file chunker
chrismccord Aug 6, 2020
ad884bf
WIP
chrismccord Aug 7, 2020
886bef0
skip sending file_data on change
mcrumm Aug 10, 2020
8388d95
Use explicit chunk based render_upload API for deterministic testing
chrismccord Aug 11, 2020
9ab27d0
Use explicit chunk based render_upload API for deterministic testing
chrismccord Aug 11, 2020
5475460
Validate and track uploads on phx-change
chrismccord Aug 18, 2020
0d6fe7e
Resolve conflicts
chrismccord Aug 18, 2020
57a2d4b
Test channel shutdown when client sends more bytes than allowed
chrismccord Aug 18, 2020
a6bdbec
Use chunk timer to protect from slowloris attack
chrismccord Aug 18, 2020
0caf588
cleanup
chrismccord Aug 19, 2020
47edd56
Close the loop on channel uploads with file consumption
chrismccord Aug 19, 2020
22369e8
Add ability to cancel uploads
chrismccord Aug 21, 2020
d4281fa
Test cancel_upload
chrismccord Aug 21, 2020
ee40500
Add consume_uploaded_entry
chrismccord Aug 24, 2020
43e6756
Add external uploaders
chrismccord Aug 25, 2020
bb0d889
Move test file
chrismccord Aug 25, 2020
ca53c12
Add external upload tests
chrismccord Aug 25, 2020
067b4d4
Allow client to report entry error
chrismccord Aug 25, 2020
a9e075a
Move upload operations to own module
chrismccord Aug 25, 2020
b785f1e
Raise when calling allow_upload on active entries
chrismccord Aug 25, 2020
f491bec
Docs
chrismccord Aug 26, 2020
1a0840d
Kill unused struct
chrismccord Aug 26, 2020
e9f16c0
touchup
chrismccord Aug 26, 2020
f70754e
docs
chrismccord Aug 26, 2020
5ce9edc
Return map of metadata on channel upload consume
chrismccord Aug 26, 2020
5df7bf1
Track files in FileList separately from input element to fix tracking…
chrismccord Aug 26, 2020
2c982d4
WIP
chrismccord Aug 27, 2020
6b7588c
Add live_img_preview
chrismccord Aug 27, 2020
d1f3b00
Touchup
chrismccord Aug 27, 2020
c53a057
Docs touches
mcrumm Aug 28, 2020
64addcc
Add guide for chunked http uploads
mcrumm Aug 28, 2020
7be81e7
Fix disabled form inputs for uploads and remove unused fields
chrismccord Aug 28, 2020
beb45f7
Add onNodeAdded dom callback
chrismccord Aug 28, 2020
80de5ec
Use ref
chrismccord Sep 18, 2020
f06997e
Prep for phoenix binary serialization support
chrismccord Oct 6, 2020
a8c51ce
Resolve conflicts
chrismccord Oct 6, 2020
0a32d22
Resolve conflicts
chrismccord Oct 6, 2020
4a45671
Fix tests
chrismccord Oct 14, 2020
8bfe9a5
Gargage collect transport when upload is complete
chrismccord Oct 14, 2020
487a3f7
Trigger entry error on channel error
chrismccord Oct 14, 2020
6a9039d
format
chrismccord Oct 15, 2020
1fc75af
Docs
chrismccord Oct 15, 2020
6930c10
Docs
chrismccord Oct 15, 2020
90a81ac
Docs
chrismccord Oct 15, 2020
a950446
Docs
chrismccord Oct 15, 2020
d0a7341
Fix docs
chrismccord Oct 15, 2020
e4d2e48
fix drag and drop options and touchup docs
chrismccord Oct 15, 2020
7abdca2
Remove uneeded binding
chrismccord Oct 16, 2020
1e6a856
Update lib/phoenix_live_view/helpers.ex
chrismccord Oct 16, 2020
c9baf78
Raise on disallow_upload for active entries
chrismccord Oct 16, 2020
4b6100c
Remove unnecessary map
chrismccord Oct 16, 2020
b1092a9
Update lib/phoenix_live_view/upload_channel.ex
chrismccord Oct 19, 2020
6be5354
Update lib/phoenix_live_view/helpers.ex
chrismccord Oct 19, 2020
9a96384
Update lib/phoenix_live_view/helpers.ex
chrismccord Oct 19, 2020
75840b0
Update lib/phoenix_live_view/test/client_proxy.ex
chrismccord Oct 19, 2020
960b06c
Fix failing test
chrismccord Oct 19, 2020
483c707
Typo
chrismccord Oct 19, 2020
405a155
Typo
chrismccord Oct 19, 2020
4dc7c91
Touchup docs
chrismccord Oct 19, 2020
f30eb8d
Touchup docs
chrismccord Oct 19, 2020
b0d7a07
Update lib/phoenix_live_view/channel.ex
chrismccord Oct 19, 2020
454baa7
Do not expose proxy_pid
chrismccord Oct 19, 2020
e700faf
Update lib/phoenix_live_view/upload_channel.ex
chrismccord Oct 19, 2020
a2669a4
typos
chrismccord Oct 19, 2020
bfd2021
Merge branch 'cm-uploads-merge' of github.com:phoenixframework/phoeni…
chrismccord Oct 19, 2020
b02474a
Fix test
chrismccord Oct 19, 2020
472db90
server uploads guide
mcrumm Oct 20, 2020
d456cfe
move external uploads into client guides
mcrumm Oct 20, 2020
f0b0ede
add file input section to form bindings guide
mcrumm Oct 20, 2020
d0fb5a7
typos
mcrumm Oct 20, 2020
6f8f3fb
Raise in_progress error in caller for better error messages
chrismccord Oct 20, 2020
55ec6e0
Update lib/phoenix_live_view/upload_channel.ex
chrismccord Oct 20, 2020
dd02973
Derive Inspect for public UploadConfig fields
chrismccord Oct 20, 2020
2fe28f5
Refactor type/ext accept matching
chrismccord Oct 20, 2020
46d870c
Update lib/phoenix_live_view.ex
chrismccord Oct 20, 2020
431505a
Precompute accept value
chrismccord Oct 20, 2020
df92b2d
Fix map construction
chrismccord Oct 20, 2020
6014562
Refactor allow_upload per feedback
chrismccord Oct 21, 2020
84a9c39
Update guides/server/uploads.md
chrismccord Oct 21, 2020
657f5ed
Fix LiveImgPreview hook
chrismccord Oct 21, 2020
4e9c8e0
Resolve conflicts
chrismccord Oct 22, 2020
91ee78b
Fix js build
chrismccord Oct 22, 2020
2aba1e5
Fix race on CI
chrismccord Oct 22, 2020
65521c4
Fix formEl lookup
chrismccord Oct 22, 2020
47cd5eb
Bump build
chrismccord Oct 22, 2020
f2aa53e
Fix tests caused by incompat node DOM apis
chrismccord Oct 23, 2020
675dd4b
Support auto_upload of file uploads and custom progress events
chrismccord Oct 28, 2020
ccb2041
Do not require form id
chrismccord Oct 29, 2020
e4418a9
Bump docs
chrismccord Nov 2, 2020
72ef657
Do not require user defined binding_prefix
chrismccord Nov 2, 2020
2850129
constants
chrismccord Nov 2, 2020
3ba0d63
Update lib/phoenix_live_view/upload_config.ex
chrismccord Nov 2, 2020
e5f9e28
Prune timer messages when raced
chrismccord Nov 2, 2020
888fe06
Update lib/phoenix_live_view.ex
chrismccord Nov 2, 2020
366fa14
Update mix.exs
chrismccord Nov 2, 2020
6ea1564
Fix spec
chrismccord Nov 2, 2020
c191922
Update test/support/endpoint.ex
chrismccord Nov 2, 2020
ead2ba1
Update test/support/endpoint.ex
chrismccord Nov 2, 2020
11a2573
Fix data shadowing in uploader and optimize LiveImgPreview
chrismccord Nov 2, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
376 changes: 364 additions & 12 deletions assets/js/phoenix_live_view.js

Large diffs are not rendered by default.

13 changes: 13 additions & 0 deletions guides/client/form-bindings.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,19 @@ requires explicitly setting the `:value` in your markup, for example:
<%= error_tag f, :password %>
<%= error_tag f, :password_confirmation %>

## File inputs

LiveView forms support [reactive file inputs](uploads.md),
including drag and drop support via the `phx-drop-target`
attribute:

<div class="container" phx-drop-target="<%= @uploads.avatar.ref %>">
...
<%= live_file_input @uploads.avatar %>
</div>

See `Phoenix.LiveView.Helpers.live_file_input/2` for more.

## Submitting the form action over HTTP

The `phx-trigger-action` attribute can be added to a form to trigger a standard
Expand Down
177 changes: 177 additions & 0 deletions guides/client/uploads-external.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@
# External Uploads

Uploads to external cloud providers, such as Amazon S3,
Google Cloud, etc., can be achieved by using the
`:external` option in [`allow_upload/3`](`Phoenix.LiveView.allow_upload/3`).

You provide a 2-arity function to allow the server to
generate metadata for each upload entry, which is passed to
a user-specified JavaScript function on the client.

Typically when your function is invoked, you will generate a
presigned URL, specific to your cloud storage provider, that
will provide temporary access for the end-user to upload data
directly to your cloud storage.

## Chunked HTTP Uploads

For any service that supports large file
uploads via chunked HTTP requests with `Content-Range`
headers, you can use the UpChunk JS library by Mux to do all
the hard work of uploading the file.

You only need to wire the UpChunk instance to the LiveView
UploadEntry callbacks, and LiveView will take care of the rest.

Install [UpChunk](https://github.com/muxinc/upchunk):

npm install --prefix assets --save @mux/upchunk

Configure your uploader on `c:Phoenix.LiveView.mount/3`:

def mount(_params, _session, socket) do
{:ok,
socket
|> assign(:uploaded_files, [])
|> allow_upload(:avatar, accept: :any, max_entries: 3, external: &presign_upload/2)}
end

Supply the `:external` option to
`Phoenix.LiveView.allow_upload/3`. It requires a 2-arity
Function that generates a signed URL where the client will
push the bytes for the upload entry.

For example, if you were using a context that provided a
[`start_session`](https://developers.google.com/youtube/v3/guides/using_resumable_upload_protocol##Start_Resumable_Session)
function, you might write something like this:

defp presign_upload(entry, socket) do
{:ok, %{"Location" => link}} =
SomeTube.start_session(%{
"uploadType" => "resumable",
"x-upload-content-length" => entry.client_size
})

{:ok, %{uploader: "UpChunk", entrypoint: link}, socket}
end

Finally, on the client-side, we use UpChunk to create an
upload from the temporary URL generated on the server and
attach listeners for its events to the entry's callbacks:

```js
import * as UpChunk from "@mux/upchunk"

let Uploaders = {}

Uploaders.UpChunk = function(entries, onViewError){
entries.forEach(entry => {
// create the upload session with UpChunk
let { file, meta: { entrypoint } } = entry
let upload = UpChunk.createUpload({ entrypoint, file })

// stop uploading in the event of a view error
onViewError(() => upload.pause())

// upload error triggers LiveView error
upload.on("error", (e) => entry.error(e.detail))

// notify progress events to LiveView
upload.on("progress", (e) => entry.progress(e.detail))

// notify complete to LiveView
upload.on("success", () => entry.done())
})
}

// Don't forget to assign Uploaders to the liveSocket
let liveSocket = new LiveSocket("/live", Socket, {
uploaders: Uploaders,
params: {_csrf_token: csrfToken}
})
```

## Direct to S3

In order to enforce all of your file constraints when
uploading to S3, it is necessary to perform a multipart form
POST with your file data.

```elixir
def mount(_params, _session, socket) do
{:ok,
socket
|> assign(:uploaded_files, [])
|> allow_upload(:avatar, accept: :any, max_entries: 3, external: &presign_upload/2)}
end

defp presign_upload(entry, socket) do
uploads = socket.assigns.uploads
bucket = "phx-upload-example"
key = "public/#{entry.client_name}"

config = %{
region: "us-east-1",
access_key_id: System.fetch_env!("AWS_ACCESS_KEY_ID"),
secret_access_key: System.fetch_env!("AWS_SECRET_ACCESS_KEY")
}

{:ok, fields} =
S3.sign_form_upload(config, bucket,
key: key,
content_type: entry.client_type,
max_file_size: uploads.avatar.max_file_size,
expires_in: :timer.hours(1)
)

meta = %{uploader: "S3", key: key, url: "http://#{bucket}.s3.amazonaws.com", fields: fields}
{:ok, meta, socket}
end
```

Here, we implemented a `presign_upload/2` function, which we passed as a captured anonymous
function to `:external`. Next, we used `ExAws.S3` to generate a presigned URL for the
upload. Lastly, we return our `:ok` result, with a payload of metadata for the client,
along with our unchanged socket. The metadata *must* contain the `:uploader` key,
specifying name of the JavaScript client-side uploader, in this case "S3".

To complete the flow, we can implement our `S3` client uploader and tell the
`LiveSocket` where to find it:

```js
let Uploaders = {}

Uploaders.S3 = function(entries, onViewError){
entries.forEach(entry => {
let formData = new FormData()
let {url, fields} = entry.meta
Object.entries(fields).forEach(([key, val]) => formData.append(key, val))
formData.append("file", entry.file)
let xhr = new XMLHttpRequest()
onViewError(() => xhr.abort())
xhr.onload = () => xhr.status === 204 ? entry.done() : entry.error()
xhr.onerror = () => entry.error()
xhr.upload.addEventListener("progress", (event) => {
if(event.lengthComputable){
let percent = Math.round((event.loaded / event.total) * 100)
entry.progress(percent)
}
})

xhr.open("POST", url, true)
xhr.send(formData)
})
}

let liveSocket = new LiveSocket("/live", Socket, {
uploaders: Uploaders,
params: {_csrf_token: csrfToken}
})
```

We define an `Uploaders.S3` function, which receives our entries. It then
performs an AJAX request for each entry, using the `entry.progress()`,
`entry.error()`, and `entry.done()` functions to report upload events
back to the LiveView. Lastly, we pass the `uploaders` namespace to the
`LiveSocket` constructor to tell phoenix where to find the uploaders
return within the external metadata.
90 changes: 90 additions & 0 deletions guides/server/uploads.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# Uploads

LiveView supports interactive file uploads with progress for
both direct to server uploads as well as external
direct-to-cloud uploads on the client.

Uploads are enabled by using `Phoenix.LiveView.allow_upload/3`
and specifying the constraints, such as accepted file types,
max file size, number of maximum selected entries, etc.
When the client selects file(s), the file metadata is
automatically validated against the `allow_upload`
specification. Uploads are populated in an `@uploads` assign
in the socket, granting reactive based templates that
automatically update with progress, error information, etc.

The complete upload flow is as follows:

## Allow uploads

You enable an upload, typically on mount, via
[`allow_upload/3`](`Phoenix.LiveView.allow_upload/3`):

```elixir
@impl Phoenix.LiveView
def mount(_params, _session, socket) do
{:ok,
socket
|> assign(:uploaded_files, [])
|> allow_upload(:avatar, accept: ~w(.jpg .jpeg), max_entries: 2)}
end
```

## Render reactive elements

Use the `Phoenix.LiveView.Helpers.live_file_input/2` file
input generator to render a file input for the upload.
The generator has full support for `multiple=true`, and the
attribute will automatically be set if `:max_entries` is
greater than 1 in the [`allow_upload/3`](`Phoenix.LiveView.allow_upload/3`) spec.

Within the template, you render each upload entry. The entry
struct contains all the information about the upload,
including progress, name, errors, etc.

For example:

```elixir
<%= for entry <- @uploads.avatar.entries do %>
<%= entry.client_name %> - <%= entry.progress %>%
<% end %>

<form phx-submit="save">
<%= live_file_input @uploads.avatar %>
<button type="submit">Upload</button>
</form>
```

Reactive updates to the template will occur as the end-user
interacts with the file input.

## Consume uploaded entries

When the end-user submits a form containing a
[`live_file_input/2`](`Phoenix.LiveView.Helpers.live_file_input/2`),
the JavaScript client first uploads the file(s) before
invoking the callback for the form's `phx-submit` event.

Within the callback for the `phx-submit` event, you invoke
the `Phoenix.LiveView.consume_uploaded_entries/3` function
to process the completed uploads, persisting the relevant
upload data alongside the form data:

```elixir
def handle_event("save", _params, socket) do
uploaded_files =
consume_uploaded_entries(socket, :avatar, fn %{path: path} ->
chrismccord marked this conversation as resolved.
Show resolved Hide resolved
dest = Path.join("priv/static/uploads", Path.basename(path))
File.cp!(path, dest)
Routes.static_path(socket, "/uploads/#{Path.basename(dest)}")
end)
{:noreply, update(socket, :uploaded_files, &(&1 ++ uploaded_files))}
end
```

> **Note**: While client metadata cannot be trusted, max file
> size validations are enforced as each chunk is received
> when performing direct to server uploads.

For more information on implementing client-side,
direct-to-cloud uploads, see the [External Uploads guide](uploads-external.md).
Loading