Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NIP-95 - Storage and Shared File #345

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
88 changes: 88 additions & 0 deletions 95.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
NIP-95 - Storage and Shared File
======
`draft` `optional` `author:frbitten`

This NIP allows users to send binary data to be stored in the relay. This data can be small pieces of information or even large files.

The intention is to combine the ease of NOSTR, with many clients being developed and a common communication protocol between all. To enable decentralized file sharing. NOSTR communication must work together with other protocols to get the best out of each technology. For lay users to be able to publish their photo, file, audio on several servers in a simple and easy way.

Nostr event
------------------
This NIP specifies the use of the `1064` event type, having in `content` the binary data that you want to be stored in Base64 format.
* `type` a string indicating the data type of the file. The MIME types format must be used (https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if you missed it in my other comment but IMO type probably belongs on kind 1065?

Ah, yes, it's already over there as an m tag.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If kind 1065 is optional, type is needed. What do you think?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's important to separate the metadadta (kind 1065) and the real data (kind 1064). To quote another part of this doc:

Another defined event is the 1065 which is used as a header for the data contained in an event 1064. This way the data can be disclosed without overloading the communications when sending a large amount of data.

I don't need a "type" here if the "m" tag is present on 1065 -- it'll be redundant. And I'm not sure 1065 events should be considered optional. Clients should have a smaller 1065 metadata event available so that they can decide whether they want to download the (likely) larger 1064 event. (Or, in the case of multi-part attachments, multiple events.)

* `decrypt` (optional) If the file is encrypted, you must indicate in the first value the algorithm used to encrypt the file and in the second value the parameters that allow the decryption of the file

This event must not be returned in generic searches. It should only be returned if specifically requested by your ID. This avoids bandwidth consumption and unnecessary overload on the relay and client.

```json
{
"id": <32-bytes lowercase hex-encoded sha256 of the the serialized event data>,
"pubkey": <32-bytes lowercase hex-encoded public key of the event creator>,
"created_at": <unix timestamp in seconds>,
"kind": 1064,
"tags": [
["decrypt",<algorithm>,<Decryption Params>],
["p", <32-bytes hex of a pubkey>, <recommended relay URL>],

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is p redundant with pubkey here? Is this just so that you can recommend a relay URL for that pubkey?

Do you want to include an example type tag here? Actually, metadata like that should probably be moved to the 1065 kind, right? Would be useful for clients to know mime type, size, dimensions, etc, before choosing whether to fetch the file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Just to reference other users, following the NIP-01 standard. It is not mandatory.

],
"content": <string with base64 data>,
"sig": <64-bytes hex of the signature of the sha256 hash of the serialized event data, which is the same as the "id" field>
}
```

Another defined event is the `1065` which is used as a header for the data contained in an event 1064. This way the data can be disclosed without overloading the communications when sending a large amount of data.

```json
{
"id": <32-bytes lowercase hex-encoded sha256 of the the serialized event data>,
"pubkey": <32-bytes lowercase hex-encoded public key of the event creator>,
"created_at": <unix timestamp in seconds>,
"kind": 1065,
"tags": [
["e",<event id 1064 you want to disclose>],
["m", <MIME type>],
["x",<Hash SHA-256>],
["size", <size of data in bytes>],
["dim", <size of data in pixels>],
["thumb", <event id that contains the data for thumb>],
["summary", <excerpt>],
["alt", <description>]
],
"content": <description>,
"sig": <64-bytes hex of the signature of the sha256 hash of the serialized event data, which is the same as the "id" field>
}
````
* `e` event id 1064 you want to disclose
* `m` a string indicating the data type of the file. The [MIME types](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types) format must be used, and they should be lowercase.
* `x` containing the SHA-256 hexencoded string of data.
* `size` (optional) size of data in bytes
* `dim` (optional) size of data in pixels in the form `<width>x<height>`
* `thumb` (optional) event id that contains the data for thumb, which can be an event 1063 or 1064
* `summary` (optional) text excerpt
* `alt` (optional) description for accessibility

Relay Behavior
---------------
Relays that use a relational database may have more difficulty implementing this NIP, but a possible solution is for this NIP not to be recorded in the database, but on disk, the file name being the event id. So it can be easily found and searched. And because it is not in the database, it does not interfere with the indexing of common events.
Copy link

@futpib futpib May 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has a caveat due to the fact that there are multiple valid base64 encodings of a same file https://medium.com/asecuritysite-when-bob-met-alice/base64-magic-8767d8e79079:

$ echo '0w==' | base64 -d | hexdump

0000000 00d3
0000001

$ echo '00==' | base64 -d | hexdump

0000000 00d3
0000001

When server and client use different base64 implementation this will result in an invalid signature when server reconstructs the event using the image file.

Possible solutions:

  1. Server has to store the original base64-encoded string
    a. Server could store a hint to be able to reconstruct the original encoding used by the client, for example last 4 characters of the original base64 string or an equivalent integer (I think a boolean should be enough but I hope someone checks this)
  2. Server should brute force valid base64 encodings of the file to find the one that matches the signature (I think there are at most two candidates but I hope someone checks this)
  3. A different encoding is used, which uniquely encodes the file
    a. A variation on base64 is proposed (or discovered if already exists) and used (like base64url, let's call it base64stable), server should reject relevant events with non-base64stable encoded content (even if it's a valid base64)
  4. Client should pad content before base64 encoding to get a unique base64 encoding (avoiding = padding from base64), server rejects relevant events with unpadded (that is padded by base64, ending with =) content

I think docs should at least mention this if this NIP is to be merged.


Another solution is for Relays that want to implement this functionality and use a No-SQL database with mongodb that already supports large documents without harming performance.

Client Behavior
---------------
The client can use this event for any type of storage it wants. The functionality of uploading an image directly through the client's interface and including it in the note that is being created is a very useful facility for the end user. In this case, 3 events would be created. Kind 1064 with the file converted to base64, Kind 1065 which is the header of this file for dissemination and Kind 1 with the post and the image.

Also, it allows the app to create image galleries (memes, animations) that can be reused countless times in different notes. As it exists in whatsapp, tegegran, etc.

Example: <https://ibb.co/Fnj5TMg>.

To do this, just select the image from the gallery (events NIP-94) and include the URL of the selected image

Suggested Use Cases
-------------------
* Provide file storage service that is quickly integrable with nostr clients.
* Create an application similar to pinterest without the need for other protocols.

References
---------------
* https://www.mongodb.com/developer/products/mongodb/storing-large-objects-and-files/
* https://www.mongodb.com/docs/manual/core/gridfs/
* https://www.linkedin.com/pulse/do-store-files-your-database-mongodbs-gridfs-thinks-otherwise-gordon/