First-class integration with Arweave

[MantisClone]

Is your feature request related to a problem? Please describe

There is a desire for first-class integration with Arweave, as stated by the OceanDAO Round 18 Tech Earmarks

The desired solution is further elaborated by the OceanDAO Tech Tributes

Describe the solution you'd like

I propose we:

1. Add a field on the 2nd publish screen for the Arweave transaction ID
2. Update the Files section of the DDO specification to include the arweave file type.

When publishing an asset:

On the publish/2 screen, under File, the user selects "+ From Arweave" similar to how the file form field looked in Ocean Commons. (text is red for visibility in the screenshot. Actual implementation would match existing market color pallet and fonts)

The user inputs the Arweave transaction ID instead of the File URL.

To verify, the Ocean Market calls the Provider fileinfo endpoint. The Provider verifies the transaction id by submitting a query to a new envvar called ARWEAVE_GATEWAY. ARWEAVE_GATEWAY could be https://arweave.net, the Arweave gateway operated by the Arweave core team.

DDO Spec

To support Arweave files, I propose we add a new file type

      {
        "type": "arweave",
        "transactionId": "cZ6j5PmPVXCq5Az6YGcGqzffYjx2JnsnlSajaHNr20w",
      },

Describe alternatives you've considered

1. The OPF could run its own Arweave Gateway instead of relying on the one provided by the Arweave core team.
2. For even greater control, The OPF could run its own Arweave node or use an Arweave node infra provider like Infura or Alchemy (but after a quick google search, I couldn't find one).
3. I considered using a decentralized Arweave gateway like AR.IO but it's still in development and not available.
4. I considered adding the ability to upload to Arweave, directly from the Ocean Market publish screen. This may be possible as a future improvement but would require additional work to ensure that the data publisher pays all costs involved. Might require a new smart contract. Typically uploading to Arweave requires AR tokens, but https://bundlr.network/ allows Arweave upload paid using currencies other than AR
5. I considered showing that an asset is stored in decentralized storage using a "Tag", but ultimately rejected this idea because tags are user-defined, so they could add the Arweave label even if the asset wasn't stored in Arweave.
6. I do not anticipate needing any changes to the compute-to-data backend to support Arweave files, given that they are accessed via a URL.

Additional context

Other discussions about integrating decentralized storage

1. People can upload data easily (and we don't store it) market#446
2. Data Locker using Estuary market#1483

[MantisClone]

I'd love to get input on this from the core team. @alexcos20 @kremalicious @mihaisc @trentmc Am I on the right track?

[T0admomo]

When I spoke with Sam from Arweave he mentioned that arweave.net was a reliable gateway. Here is the repo from Arweave for setting up an arweave gateway. https://github.com/ArweaveTeam/gateway

[MantisClone]

Based on feedback in Ocean Protocol Discord #core-tech channel, I realized some publishers may not want to reveal that their files are stored in decentralized storage like Arweave. In light of this, I decided not to change the services.files structure in the DDO spec. As a consequence, I also cannot implement the proposed "Stored in Arweave" indicators when viewing assets.

For posterity, here is the original PR description. I've updated the issue description to show the updated design.

Is your feature request related to a problem? Please describe

Today, I believe it is possible to publish an asset on Ocean Market using an Arweave URL like: https://arweave.net/<transaction id>. But after publishing, the user has no indication that an asset is stored in decentralized storage unless the asset publisher states it in their description.

There is a desire for first-class integration with Arweave, as stated by the OceanDAO Round 18 Tech Earmarks

The desired solution is further elaborated by the OceanDAO Tech Tributes

Describe the solution you'd like

I propose we:

1. Add a field on the 2nd publish screen for the Arweave transaction ID
2. Add indicators on the asset browse and asset details screens to show that an asset is stored in Arweave.
3. Update the [DDO specification] to make file type information available to the ocean market
   (https://github.com/oceanprotocol/docs/blob/main/content/concepts/did-ddo.md).

When publishing an asset:

On the publish/2 screen, under File, the user selects "+ From Arweave" similar to how the file form field looked in Ocean Commons. (text is red for visibility in the screenshot. Actual implementation would match existing market color pallet and fonts)

The user inputs the Arweave transaction ID instead of the File URL.

To verify, the Ocean Market calls the Provider fileinfo endpoint. The Provider verifies the transaction id by submitting a query to a new envvar called ARWEAVE_GATEWAY. ARWEAVE_GATEWAY could be https://arweave.net, the Arweave gateway operated by the Arweave core team.

When viewing an asset:

I'd like there to be an indicator that shows the file is stored in Arweave decentralized storage. Something like the following

DDO Spec

EDIT: This section has been fully overhauled. Previously I proposed adding a services.storedIn key to the DDO Spec, but later realized this clashed with the existing services.files key which already contains type info (URL, IPFS, Arweave...).

Currently, the DDO spec specifies services.files is an encrypted string. When decrypted it contains an array of file objects which differ in content depending on where the file "type" (e.g. a "url" file object contains the URL and HTTPS method, an "ipfs" file object contains the IPFS hash)

I propose that we change services.files from a single encrypted string to an array of file objects where the type is plaintext and the URL or Hash is encrypted. This would make the file type available to the ocean market and would enable the "Stored in Arweave" indicator described in the previous section. This would also allow the ocean market to see how many files are associated with a given service.

For example:

{
  "services": {
    "files": [
      {
        "type": "url",
        "encrypted_accessor": <encrypted string, when decrypted contains object>:
          {
            "url": "https://hello.world",
            "method": "GET",
          }
      },
      {
        "type": "ipfs",
        "encrypted_accessor": <encrypted string, when decrypted contains object>: 
          {
            "hash": "QmWWQSuPMS6aXCbZKpEjPHPUZN2NjB3YrhJTHsV4X3vb2t",
          }
      },
}

Furthermore, to support Arweave files, I propose we add a new file type

      {
        "type": "arweave",
        "encrypted_accessor": <encrypted string, when decrypted contains object>: 
          {
            "transactionId": "cZ6j5PmPVXCq5Az6YGcGqzffYjx2JnsnlSajaHNr20w",
          }
      },

P.S. I recognize this DDO spec change would affect components and tests throughout the stack, including market, ocean.js, ocean.py, provider, and aquarius.

Describe alternatives you've considered

1. The OPF could run its own Arweave Gateway instead of relying on the one provided by the Arweave core team.
2. For even greater control, The OPF could run its own Arweave node or use an Arweave node infra provider like Infura or Alchemy (but after a quick google search, I couldn't find one).
3. I considered using a decentralized Arweave gateway like AR.IO but it's still in development and not available.
4. I considered adding the ability to upload to Arweave, directly from the Ocean Market publish screen. This may be possible as a future improvement but would require additional work to ensure that the data publisher pays all costs involved. Might require a new smart contract. Typically uploading to Arweave requires AR tokens, but https://bundlr.network/ allows Arweave upload paid using currencies other than AR
5. I considered showing that an asset is stored in decentralized storage using a "Tag", but ultimately rejected this idea because tags are user-defined, so they could add the Arweave label even if the asset wasn't stored in Arweave.
6. I do not anticipate needing any changes to the compute-to-data backend to support Arweave files, given that they are accessed via a URL.

Additional context

Other discussions about integrating decentralized storage

1. People can upload data easily (and we don't store it) market#446
2. Data Locker using Estuary market#1483

[mihaisc]

Taking a quick glance it looks ok from a market point of view. Just to make sure i understand correct, in the market we just need the a transaction id and then the provider does the magic

[MantisClone]

Taking a quick glance it looks ok from a market point of view. Just to make sure i understand correct, in the market we just need the a transaction id and then the provider does the magic

Thanks for taking a look @mihaisc! Yes, you're correct.