Support for range request queries #526

scouten · 2017-03-06T05:52:04Z

As mentioned in #523, we would like to serve some light static video content through a Phoenix-based web site and are encountering the issue described in http://stackoverflow.com/questions/36576801/serving-http-range-request-with-phoenix.

This PR adds part of the range request protocol defined in RFC 7233 to Plug.Static.

This implementation supports requests specifying a single range of bytes, which appears to be sufficient to enable video playback in Safari. A client may request multiple byte ranges in the same request, but I am not proposing to implement that in this PR.

Submitting now for preliminary / architectural review. IMHO it is not yet ready for merge.

My to do list (I will probably get back to this later in the week):

Double-check implementation against RFC 7233 specs for compliance
Review tests for error conditions
Check code coverage
(Possibly?) Add support for cache mechanisms

sourcelevel-bot · 2017-03-06T05:53:03Z

lib/plug/static.ex

+    encoding = file_encoding(conn, path, false, false)
+    serve_range(encoding, range, segments, options)
+  end
+  defp serve_range(_conn, _path, _segments, range, _gzip?, _brotli?, _options) do


Function takes too many parameters (arity is 7, max is 5).

sourcelevel-bot · 2017-03-06T05:53:04Z

lib/plug/static.ex

+    encoding = file_encoding(conn, path, gzip?, brotli?)
+    serve_static(encoding, segments, options)
+  end
+  defp serve_range(conn, path, segments, [range], _gzip?, _brotli?, options) do


Function takes too many parameters (arity is 7, max is 5).

sourcelevel-bot · 2017-03-06T05:53:04Z

lib/plug/static.ex

@@ -177,6 +181,100 @@ defmodule Plug.Static do
    h in only or match?({0, _}, prefix != [] and :binary.match(h, prefix))
  end

+  defp serve_range(conn, path, segments, [], gzip?, brotli?, options) do


Function takes too many parameters (arity is 7, max is 5).

Not sure yet how to adjust send_range/7.

scouten · 2017-04-21T00:57:51Z

@josevalim et al: I believe this is ready for review and merge.

josevalim · 2017-06-06T12:49:23Z

lib/plug/static.ex

+    raise InvalidRangeError
+  end
+
+  @byte_range_pattern ~r/^\s*bytes=([0-9]+)?-([0-9]+)?\s*$/


Please parse the bytes accordingly instead of relying on regexes. See the functions in Plug.Conn.Utils that can help you parse params. If necessary, add new functions there.

josevalim · 2017-06-06T12:49:57Z

lib/plug/static.ex

+
+  defp start_and_end([_, range_start], file_size)
+    when is_binary(range_start)
+  do


Please make the code style consistent with the remaining of the Plug codebase. In this particular case, everything can fit a single line.

josevalim · 2017-06-06T12:53:46Z

lib/plug/static.ex

+  defp send_range(conn, path, range_start, range_end, file_size, segments, options) do
+    %{headers: headers} = options
+    length = (range_end - range_start) + 1
+    content_type = segments |> List.last |> MIME.from_path


You need to support custom content types, as here: https://github.com/scouten/plug/blob/a20a9a45aa04e86291ef6e67cd6b5390ccfba8ce/lib/plug/static.ex#L288-L289

Please make sure to add a test for this too.

josevalim · 2017-06-06T12:54:18Z

test/plug/static_test.exs

+    test "returns 416 if range is contains non-integers" do
+      exception = assert_raise Plug.Static.InvalidRangeError,
+                               "invalid range for static asset",
+        fn ->


Please start fn -> in the previous line and outdent the whole function 2 spaces. The same for the examples below.

josevalim · 2017-06-06T12:55:31Z

Sorry for the delay in reviewing this. I have added some comments.

josevalim · 2017-06-06T12:56:53Z

lib/plug/static.ex

+  @byte_range_pattern ~r/^\s*bytes=([0-9]+)?-([0-9]+)?\s*$/
+
+  defp serve_range({:ok, conn, file_info, path}, range, segments, options) do
+    file_size = elem(file_info, 1)


Use record matching: file_info(size: size) = file_info

josevalim · 2017-06-06T13:00:53Z

lib/plug/static.ex

+      :entire_file ->
+        serve_static({:ok, conn, file_info, path}, segments, options)
+      :invalid ->
+        raise InvalidRangeError


Is raising the reasonable thing to do here? You said we don't support all syntaxes, so maybe we should rather send the whole file instead of crashing.

I can't find this exact scenario in the spec but it seems like, generally, if there's a range request the server does not understand the header should be ignored.

Honestly, reading the spec, I can find support for both approaches:

Section 3.1:

A server that supports range requests MAY ignore or reject a Range header field that consists of more than two overlapping ranges, or a set of many small ranges that are not listed in ascending order, since both are indications of either a broken client or a deliberate denial-of-service attack (Section 6.1).

or (later in same section):

If all of the preconditions are true, the server supports the Range header field for the target resource, and the specified range(s) are invalid or unsatisfiable, the server SHOULD send a 416 (Range Not Satisfiable) response.

Section 4.4:

The 416 (Range Not Satisfiable) status code indicates that none of the ranges in the request's Range header field (Section 3.1) overlap the current extent of the selected resource or that the set of ranges requested has been rejected due to invalid ranges or an excessive request of small or overlapping ranges.

For byte ranges, failing to overlap the current extent means that the first-byte-pos of all of the byte-range-spec values were greater than the current length of the selected representation. When this status code is generated in response to a byte-range request, the sender SHOULD generate a Content-Range header field specifying the current length of the selected representation (Section 4.2).

I think the argument being made here is that returning the entire file may be "friendlier" than returning a 416, and I could probably go along with that. I'll put it in a separate commit within this PR so we can revert it if you disagree.

josevalim · 2017-06-06T13:00:59Z

lib/plug/static.ex

+    serve_range(encoding, range, segments, options)
+  end
+  defp serve_range(_conn, _path, _segments, _range, _options) do
+    raise InvalidRangeError


Is raising the reasonable thing to do here? You said we don't support all syntaxes, so maybe we should rather send the whole file instead of crashing.

scouten · 2017-06-12T00:34:29Z

@josevalim @ericmj this feedback all seems reasonable. I'll work on this over the next couple of weeks and send you a ping when I think it's ready for re-review.

scouten · 2017-06-12T00:57:34Z

Remaining issues:

Code format concerns
Parse bytes w/o regex
Support custom content type
Record matching in serve_range
Decide how to respond to feedback about invalid range queries

…n 206 responses.

scouten · 2017-06-12T03:12:10Z

@josevalim @ericmj Ping. I think this is ready for re-review.

ericmj · 2017-06-12T10:03:24Z

lib/plug/static.ex

+
+    conn
+    |> put_resp_header("content-type", content_type)
+    |> put_resp_header("accept-ranges", "bytes")


This header should be set for all responses from Plug.Static, no?

Also, do we not support caching or compression for range requests?

@ericmj I'm open to working up a variant that supports caching, but I don't think I have the time or knowledge to properly implement compression in this context. Given that the primary use case that I'm familiar with (video playback) applies to content that is already compressed, I'm not sure that adding more compression makes sense.

ericmj · 2017-06-12T10:06:34Z

lib/plug/static.ex

+    file_info(size: file_size) = file_info
+
+    parsed_range = range
+      |> parse_range


Add parenthesis to all function calls.

josevalim · 2017-06-14T17:42:13Z

Also paging @TheSquad which implemented a plug that does this.

scouten · 2017-06-14T21:47:59Z

FYI this is derived from @TheSquad's plug, but ties in here so as to provide the same guarantees about access, etc., that the rest of plug_static offers.

I'll work on addressing the new feedback from @ericmj in the next couple of days.

scouten · 2017-06-15T04:05:34Z

OK, I've added parens and accept-ranges header as requested.

Still looking into caching implementation.

josevalim · 2017-06-15T07:32:08Z

lib/plug/static.ex

+  defp check_bounds({range_start, range_end}, _file_size), do: {range_start, range_end}
+
+  defp to_integer(str) do
+    {num, _} = Integer.parse(str)


This can return :error and that would cause an exception. I would attempt to write it like this:

defp start_and_end("-" <> rest, file_size) do case Integer.parse(rest) do {last, ""} -> {file_size - last, file_size - 1} _ -> :error end end defp start_and_end(range, file_size) do case Integer.parse(range) do {first, "-"} -> {first, file_size - 1} {first, "-" <> rest} -> case Integer.parse(rest) do {last, ""} -> {first, last} _ -> :error end _ -> :error end end

and then in serve_range you could drive through all of those decisions:

with %{"bytes" => bytes} <- Plug.Conn.Utils.params(range), {first, last} <- start_and_end(bytes, file_size), :ok <- check_bounds(first, last) do send_range(conn, path, range_start, range_end, file_size, segments, options) else _ -> serve_static({:ok, conn, file_info, path}, segments, options) end

Note it uses a check_bounds implementation that returns :ok or :error.

You can probably add a test that makes sure it doesn't crash if someone sends a bad range such as 00-FF.

…entire file instead.

scouten · 2017-06-17T23:47:17Z

@josevalim @ericmj Ping. Ready for re-review. There was a fairly substantial refactor since prior versions to make caching work properly with range requests.

ericmj · 2017-06-18T21:37:16Z

Thank you @scouten! This is a great addition.

scouten · 2017-06-18T21:37:41Z

@josevalim @ericmj thanks for your patient reviews and thanks for accepting this contribution!

scouten added 3 commits March 5, 2017 20:43

Add initial set of tests.

84a40c9

Initial draft of range query support.

b93e884

Fix a warning and tighten down a couple of error conditions.

0274432

sourcelevel-bot bot reviewed Mar 6, 2017

View reviewed changes

scouten added 4 commits March 5, 2017 21:58

Reduce parameters on serve_range/7 (now 5) to make Ebert happy.

b4671e7

Not sure yet how to adjust send_range/7.

Fix typo in variable name.

7939b56

Improve test names.

ceb7525

Merge branch 'master' into range-request

a20a9a4

scouten mentioned this pull request Apr 28, 2017

Warnings in Elixir 1.4 TheSquad/plug_range#5

Closed

josevalim reviewed Jun 6, 2017

View reviewed changes

scouten added 2 commits June 11, 2017 17:35

Merge branch 'master' into range-request

8da24e6

Address code-format issues.

9685ce7

scouten added 6 commits June 11, 2017 18:09

More code-format consistency.

5692f99

Parse byte range without regex.

037c18c

Use record matching to extract file size from file_info record.

d0929ac

Handle custom types.

e772bd0

If unable to parse range header, just return the entire file.

5171110

Test that we're sending "accept-ranges" and "content-range" headers o…

63e4999

…n 206 responses.

Tighten up whitespace among check_bounds implementations.

63fecad

ericmj reviewed Jun 12, 2017

View reviewed changes

scouten added 2 commits June 14, 2017 21:03

Add parens on parse_range pipe stage.

c75e89c

Send accept-ranges header on request w/o range query.

4466e5f

elixir-plug deleted a comment from sourcelevel-bot bot Jun 15, 2017

josevalim reviewed Jun 15, 2017

View reviewed changes

scouten added 4 commits June 17, 2017 15:33

Merge branch 'master' into range-request

4bd6a6a

Add () to function calls in test pipe stages.

a773077

Don't crash when non-integers are passed in byte range strings. Send …

66fa628

…entire file instead.

Refactor to allow caching to apply to range requests.

54deba8

josevalim approved these changes Jun 18, 2017

View reviewed changes

ericmj approved these changes Jun 18, 2017

View reviewed changes

ericmj merged commit 01df787 into elixir-plug:master Jun 18, 2017

scouten deleted the range-request branch June 18, 2017 21:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for range request queries #526

Support for range request queries #526

scouten commented Mar 6, 2017 •

edited

Loading

sourcelevel-bot bot Mar 6, 2017

sourcelevel-bot bot Mar 6, 2017

sourcelevel-bot bot Mar 6, 2017

scouten commented Apr 21, 2017

josevalim Jun 6, 2017

josevalim Jun 6, 2017

josevalim Jun 6, 2017

josevalim Jun 6, 2017

josevalim Jun 6, 2017

josevalim commented Jun 6, 2017

josevalim Jun 6, 2017

josevalim Jun 6, 2017

ericmj Jun 6, 2017

scouten Jun 12, 2017

josevalim Jun 6, 2017

scouten commented Jun 12, 2017

scouten commented Jun 12, 2017 •

edited

Loading

scouten commented Jun 12, 2017 •

edited

Loading

ericmj Jun 12, 2017

ericmj Jun 12, 2017

scouten Jun 14, 2017

ericmj Jun 12, 2017

josevalim commented Jun 14, 2017

scouten commented Jun 14, 2017 •

edited

Loading

scouten commented Jun 15, 2017

josevalim Jun 15, 2017 •

edited

Loading

josevalim Jun 15, 2017

scouten commented Jun 17, 2017

ericmj commented Jun 18, 2017

scouten commented Jun 18, 2017

Support for range request queries #526

Support for range request queries #526

Conversation

scouten commented Mar 6, 2017 • edited Loading

sourcelevel-bot bot Mar 6, 2017

Choose a reason for hiding this comment

sourcelevel-bot bot Mar 6, 2017

Choose a reason for hiding this comment

sourcelevel-bot bot Mar 6, 2017

Choose a reason for hiding this comment

scouten commented Apr 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josevalim commented Jun 6, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scouten commented Jun 12, 2017

scouten commented Jun 12, 2017 • edited Loading

scouten commented Jun 12, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josevalim commented Jun 14, 2017

scouten commented Jun 14, 2017 • edited Loading

scouten commented Jun 15, 2017

josevalim Jun 15, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scouten commented Jun 17, 2017

ericmj commented Jun 18, 2017

scouten commented Jun 18, 2017

scouten commented Mar 6, 2017 •

edited

Loading

scouten commented Jun 12, 2017 •

edited

Loading

scouten commented Jun 12, 2017 •

edited

Loading

scouten commented Jun 14, 2017 •

edited

Loading

josevalim Jun 15, 2017 •

edited

Loading