Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add files to IPFS during publish flow #191

Merged
merged 42 commits into from Oct 14, 2019

Conversation

@kremalicious
Copy link
Member

kremalicious commented Sep 4, 2019

Highly WIP

  • support for adding a file from an ipfs:// url (#192)
  • support for adding a file to IPFS during publish flow
  • store ipfs:// url in asset DDO
  • integrate IPFS in all the tests
  • Closes #187
  • Part of oceanprotocol/ocean#363

Screen Shot 2019-09-18 at 12 52 56

Screen Shot 2019-09-18 at 12 53 47

Screen Shot 2019-09-18 at 13 26 15

Screen Shot 2019-09-18 at 12 56 25


Holds different prototypes, each prototype as their own React Hook:

1. IPFS node in browser, remote gateway

  1. When IPFS form is opened, spawn a local IPFS node in the browser. If form is closed, stop IPFS node
  2. When file is selected from file system, add the file to local IPFS node, returning the CID
  3. Ping the new CID via gateway.ipfs.io url to make it globally available
  4. Pass new ipfs.io url to file URL checker to get file attributes
  5. Pass new File info to publish form state
  6. Close IPFS form, which will also stop IPFS node

Untitled

Pro

  • super simple, all happens client-side
  • adding to local IPFS node is instant

Con

  • distribution from local node to global (by pinging gateway url) can take a long time, and depends on the user's internet connection combined with local IPFS node being connected to peers
  • we can't display any progress for that process
  • having the file globally available before passing it to our file checker is required

2. Remote IPFS node & gateway: ipfs.infura.io

  1. When IPFS form is opened, ping remote API of a 3rd party IPFS node & gateway (ipfs.infura.io), create IPFS instance from it. If form is closed, destroy IPFS instance
  2. When file is selected from file system, add the file to remote IPFS API, returning the CID
  3. Ping the new CID via gateway url to make it globally available
  4. Pass new ipfs.io url to file URL checker to get file attributes
  5. Pass new File info to publish form state
  6. Close IPFS form

Untitled

Pro

  • file is globally available almost instantly after being added to IPFS node

Con

  • adding to remote node can take a while
  • ipfs.infura.io rejects too large payloads (exact restriction is undocumented) and there seems to be no way to get around that, hence not feasible for large files
  • adding file depends on user's internet connection, but exact progress can be displayed to user

3. Remote IPFS node & gateway: ipfs.oceanprotocol.com

Requires oceanprotocol/atlantic#91

Developing...

Flow is essentially like prototype 2, but requests go to ipfs.oceanprotocol.com.

Pro

  • large files can be added
  • we control how IPFS node & gateway can be used
  • donating a IPFS node & public gateway is the logical nice thing to do

Con

  • adding file depends on user's internet connection, but exact progress can be displayed to user
@kremalicious kremalicious requested review from oceanprotocol/core-dev as code owners Sep 4, 2019
@kremalicious kremalicious force-pushed the feature/ipfs branch from 0b3f41f to a780820 Sep 4, 2019
@JohnBDB JohnBDB referenced this pull request Sep 4, 2019
5 of 5 tasks complete
@kremalicious kremalicious force-pushed the feature/ipfs branch 2 times, most recently from 8edbcca to a54b225 Sep 4, 2019
@kremalicious kremalicious force-pushed the feature/ipfs branch from c49c620 to 820f8e4 Sep 5, 2019
@kremalicious kremalicious referenced this pull request Sep 6, 2019
11 of 11 tasks complete
@kremalicious kremalicious force-pushed the feature/ipfs branch 6 times, most recently from 01b944c to 0c27b38 Sep 10, 2019
@oceanprotocol oceanprotocol deleted a comment Sep 11, 2019
@kremalicious kremalicious force-pushed the feature/ipfs branch from 7094bcf to 5af40c3 Sep 18, 2019
@oceanprotocol oceanprotocol deleted a comment Sep 18, 2019
@kremalicious

This comment has been minimized.

Copy link
Member Author

kremalicious commented Sep 18, 2019

Some feature design decisions along the way:

Single file only

While both, our drop zone and IPFS, supports adding multiple files in one go, it proved to be tricky to get upload status of individual files. So for the time being, the IPFS drop zone is locked to one file.

File names

While every file on IPFS has a url ipfs://CID, using this direct file url results in the file name getting lost. So storing that direct url in the DDO will result in a octet-stream blob being downloaded during consume flow, with no way of knowing what file type it is.

I was able to work around that by getting the original name and storing it in our DDO since squid-js supports that in most recent version. But a more future-proof and Ocean-independent way is to wrap a file in a directory on IPFS, which preserves the file's original file name.

By using this folder trick we end up with a url like ipfs://CID/FILENAME.pdf where the CID is the hash of the folder, not the file. And then we store this url as the file url in the DDO.

@kremalicious kremalicious force-pushed the feature/ipfs branch 2 times, most recently from 158712f to da949bb Sep 18, 2019
Copy link

ssallam left a comment

LGTM.

@oceanprotocol oceanprotocol deleted a comment Sep 19, 2019
@oceanprotocol oceanprotocol deleted a comment Sep 20, 2019
@kremalicious kremalicious force-pushed the feature/ipfs branch from bbcadf1 to f8bbcf6 Sep 20, 2019
@oceanprotocol oceanprotocol deleted a comment Sep 24, 2019
@kremalicious kremalicious force-pushed the feature/ipfs branch 2 times, most recently from 0bad2ce to 3bf821a Oct 7, 2019
@kremalicious

This comment has been minimized.

Copy link
Member Author

kremalicious commented Oct 11, 2019

Remaining issue is the communication to our IPFS node's API. We have both, the gateway and the node API exposed on port 443, and Nginx then does a proxy pass for the /api/* route under 5001.

Nginx and CORS are correctly setup and when doing this from terminal everything works as expected:

curl -X POST https://ipfs.dev-ocean.com/api/v0/version

In the UI, js-ipfs-http-api does exactly the same API call with ipfs.version() but this returns:

Screen Shot 2019-10-11 at 10 35 13

When doing the same thing against my own IPFS node setup on a VPS:

Screen Shot 2019-10-11 at 10 39 14

giphy-1

@oceanprotocol oceanprotocol deleted a comment Oct 11, 2019
kremalicious added 20 commits Sep 10, 2019
* only allow single file for now
* async file content reading
* output list of files by default in Dropzone component
@kremalicious kremalicious force-pushed the feature/ipfs branch from 80749af to 59ede97 Oct 14, 2019
@oceanprotocol oceanprotocol deleted a comment Oct 14, 2019
@codeclimate

This comment has been minimized.

Copy link

codeclimate bot commented Oct 14, 2019

Code Climate has analyzed commit 4fb47c0 and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 86.1% (50% is the threshold).

This pull request will bring the total coverage in the repository to 76.2% (2.9% change).

View more on Code Climate.

@kremalicious kremalicious referenced this pull request Oct 14, 2019
@kremalicious kremalicious merged commit 54a4ba5 into master Oct 14, 2019
5 of 6 checks passed
5 of 6 checks passed
Codacy/PR Quality Review Hang in there, Codacy is reviewing your Pull request.
Details
Travis CI - Branch Build Passed
Details
Travis CI - Pull Request Build Passed
Details
codeclimate 2 fixed issues
Details
codeclimate/diff-coverage 86% (50% threshold)
Details
codeclimate/total-coverage 76% (2.9% change)
Details
@kremalicious kremalicious deleted the feature/ipfs branch Oct 14, 2019
santteegt added a commit to decentraminds/mktpl that referenced this pull request Oct 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.