Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gateway Improvements: Streaming, Conditional and Range Requests #1989

Merged
merged 3 commits into from May 8, 2019

Conversation

Projects
3 participants
@lidel
Copy link
Member

commented Apr 11, 2019

Part of an effort to run embedded js-ipfs in Brave 馃 ipfs-shipyard/ipfs-companion#716

This PR:

  • Fixes code responsible for streaming responses and makes the streaming actually work by telling the payload compression stream to flush its content on every read()
    Previous version was buffering entire thing in Hapi's compressor memory.
    See hapijs/hapi#3599 and hapijs/hapi@c49b9c8 for more details.
  • Improves content-type detection based on the beginning of the stream. Gateway is peeking at first fileType.minimumBytes bytes and sets content-type header.

Added 2019-04-29:

  • Switched from deprecated hapi and joi to @hapi/hapi and @hapi/joi
  • Added support for Conditional Requests (RFC7232)
    • Returning 304 Not Modified if If-None-Match is a CID matching Etag
    • Added Last-Modified to /ipfs/ responses (improves client-side caching)
    • Always returning 304 Not Modified If-Modified-Since for immutable /ipfs/
  • Added support for Byte Range Requests (rfc7233#section-2.1)
  • Added support for ?filename= parameter (improves downloads of raw )

@lidel lidel requested review from alanshaw and hugomrdias Apr 11, 2019

@ghost ghost assigned lidel Apr 11, 2019

@ghost ghost added the in progress label Apr 11, 2019

lidel added a commit to ipfs-shipyard/ipfs-companion that referenced this pull request Apr 12, 2019

fix(brave): robust ipfs.cat + content-type sniff
This applies cherry-picked patches from:
ipfs/js-ipfs#1989
ipfs/js-ipfs#1950
and solves stream issues on page refresh.

Content-type sniffing is now done over a meaningful amount of bytes
instead of arbitrary number.

@lidel lidel referenced this pull request Apr 12, 2019

Closed

Support chrome.sockets.* APIs #664

4 of 4 tasks complete
@hugomrdias
Copy link
Collaborator

left a comment

LGTM

@lidel lidel force-pushed the fix/streaming-compressed-payload branch from c0838d8 to 373f69e Apr 18, 2019

@lidel

This comment has been minimized.

Copy link
Member Author

commented Apr 18, 2019

@alanshaw rebased this PR against latest master and CI no longer fails due to libp2p

@lidel lidel referenced this pull request Apr 22, 2019

Open

Embedded JS-IPFS in Brave #716

17 of 38 tasks complete

@lidel lidel changed the title fix(gateway): streaming compressed payload Gateway Improvements: Streaming, Conditional and Range Requests Apr 26, 2019

@lidel lidel requested a review from hugomrdias Apr 26, 2019

@lidel

This comment has been minimized.

Copy link
Member Author

commented Apr 26, 2019

As this PR was not merged and I had more improvements built on top of it, I've just pushed those additional improvements here and updated description to reflect what changed.

Note: We should review & merge this before we add support for /ipns/ to the Gateway, as there are caching nuances around mutable-vs-immutable addressed by this PR.

Question: If I wanted to add interop (go-vs-js) tests for relevant HTTP headers, which repo should I use?

lidel added a commit to ipfs-shipyard/ipfs-companion that referenced this pull request Apr 26, 2019

feat(brave): conditional and range requests
Switching to js-ipfs from ipfs/js-ipfs#1989
for next beta release

@lidel lidel referenced this pull request Apr 26, 2019

Merged

feat: conditional and range requests in Brave #717

8 of 8 tasks complete

lidel added some commits Apr 3, 2019

fix(gateway): streaming compressed payload
This change simplifies code responsible for streaming response
and makes the streaming actually work by telling the payload compression
stream to flush its content on every read().
(previous version was buffering entire thing in Hapi's compressor memory)

We also do content-type detection based on the beginning of the stream
by peeking at first `fileType.minimumBytes` bytes.

License: MIT
Signed-off-by: Marcin Rataj <lidel@lidel.org>
feat(gateway): range and conditional requests
- Switched from deprecated `hapi` and `joi` to `@hapi/hapi` and `@hapi/joi`
- Added support for Conditional Requests (RFC7232)
  - Returning `304 Not Modified` if `If-None-Match` is a CID matching `Etag`
  - Added `Last-Modified` to `/ipfs/` responses (improves client-side caching)
  - Always returning `304 Not Modified`
    when `If-Modified-Since` is present for immutable `/ipfs/`
- Added support for Byte Range requests (RFC7233, Section-2.1)
- Added support for `?filename=` parameter (improves downloads of raw cids)

License: MIT
Signed-off-by: Marcin Rataj <lidel@lidel.org>

@lidel lidel force-pushed the fix/streaming-compressed-payload branch from 17712a4 to 373eedc May 6, 2019

@lidel

This comment has been minimized.

Copy link
Member Author

commented May 6, 2019

Rebased on top of #2015, decreasing the surface of this PR.
@alanshaw should be easier to review 鈥 mind taking a look ?

@lidel lidel referenced this pull request May 7, 2019

Open

[WIP] feat: add HTTP Gateway support for /ipns/ paths #2020

1 of 6 tasks complete
refactor: remove redundant try/catch
License: MIT
Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>

@ghost ghost assigned alanshaw May 8, 2019

lidel added a commit to ipfs-shipyard/ipfs-companion that referenced this pull request May 8, 2019

feat(brave): Gateway with IPNS+DNSLink+HAMT
tl;dr opening /ipns/tr.wikipedia-on-ipfs.org/wiki/Mars.html works

Switched to js-ipfs with cherry-picked changes from:
ipfs/js-ipfs#2020
ipfs/js-ipfs#1989
ipfs/js-ipfs#2002
ipfs/js-ipfs-http-response#22
ipfs/js-ipfs-mfs#48

@alanshaw alanshaw merged commit 48a8e75 into master May 8, 2019

1 of 2 checks passed

Travis CI - Branch Build Failed
Details
Travis CI - Pull Request Build Passed
Details

@alanshaw alanshaw deleted the fix/streaming-compressed-payload branch May 8, 2019

@ghost ghost removed the in progress label May 8, 2019

@alanshaw alanshaw referenced this pull request May 9, 2019

Closed

鈿★笍 v0.36.0 RELEASE 馃殌 #2024

43 of 44 tasks complete

@momack2 momack2 added this to Done in ipfs/js-ipfs May 10, 2019

@momack2 momack2 added this to Done in ipfs/js-waffle May 10, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can鈥檛 perform that action at this time.