Nostr interoperability

[burdiyan]

I'd like to bring up a discussion on some of the technical choices of Nostr, and their impact on various areas, especially interoperability with other systems, which I believe is an important factor for an open protocol like Nostr.

Regarding interoperability, I'd like to talk mostly about IPFS. I know about your (@fiatjaf) hatred of IPFS, and I won't be trying to prove you wrong or anything (some of your complaints are valid) 😀. But I do feel that Nostr could have been built on many of the primitives IPFS provides (except the P2P component), without losing any of its spirit or simplicity.

Also, we at Mintter are using IPFS, and we'd like to integrate with Nostr as well. I believe that both things could work together much more smoothly.

IPLD And Content-Addressability

The premise of IPFS is content-addressability, and Nostr has some degree of content-addressability as well - event IDs are derived from the content of the event. But the way it works in Nostr is totally incompatible with IPFS.

The most basic thing in IPFS is the mapping of hashes to opaque bytes. So anyone can easily verify the hash without manipulating or understanding anything about those bytes. But in Nostr, all actors must understand the structure of the event, and to verify the event ID you have to manipulate the JSON data into some specific form to verify the hash (converting JSON object into an array, prepend 0, etc.).

I believe that Nostr could still be Nostr, if the event ID was derived from the exact bytes of the event, including the signature. This would also make Nostr events perfect IPFS blocks, and would be much easier to implement the validation for anyone (just hash the bytes and verify the hash).

IPLD could help a lot with hashing and signing, because IPLD defines the canonical encoding of data (be it JSON, CBOR, or anything else), so there’s no need to define any extra rules on how to manipulate the data to validate the hash (no need to convert JSON object into a JSON array with strict requirements on how to order fields).

CIDs and IPLD links could also help expressing the “linking” between different Nostr events more clearly (the e tag or any other tag that points to an ID of another event).

Reusing More Standards

In the Nostr readme, you say that “reusing a standard is always better than trying to get people in a new one”. But IMO pretty much everything in Nostr feels so custom, bespoke, and as if deliberately made to avoid using any existing standard 🙂

Don’t think I’m blaming you, we at Mintter are not using many of these standards ourselves yet, but we are looking into it, and we’d like to be using them more extensively. In the end “moving the needle” is much more important than ticking some compliance checkbox, but at the same time, it wouldn’t hurt to be more compatible with other systems.

Some of the existing standards that could be useful for Nostr:

1. UCAN for delegated signing. It's an emerging spec, and I myself have some concerns about it. But it could be used for the same purposes as NIP-26.
2. WebFinger for mapping of human-friendly IDs to public keys. This is used by BlueSky's AT Protocol, by Mastodon, and probably someone else. Using this instead of NIP-05 (which is very similar to WebFinger anyway), Nostr could be much more interoperable with these other systems. The same username could maybe even reused between systems.
3. IPLD. As already mentioned, this could help a lot with hashing and signing payloads, without having to manipulate JSON object into some other forms to hash and verify. IPLD could also help evolving the protocol, e.g. adding support for binary encoding at some point, for efficiency (although I personally would be happy if Nostr was using binary encoding from the beginning).
4. Ed25519 and other forms of public keys. Many apps are using Ed25519 private keys for their identity mechanism, and it's very widely adopted standard in general on the web. I know that you could derive different keys from the same seed or mnemonic words, but letting people directly share their public key identity among apps could make interoperability much easier.

From all the mentioned things, I believe the most impactful thing for interoperability would be using content-addressability in a more general way. I.e. having a mapping from hashes (event IDs) to the exact bytes events are serialized to. This is probably the most "breaking" idea for Nostr among all the mentioned things, but considering the adoption Nostr is getting, there would probably be no better time than now to change the fundamentals.

[fiatjaf]

Reusing a standard is good if that brings any interoperability benefit, otherwise it is just a waste. For Nostr to work with any other system, a full-fledged custom from ground-up mechanism would have to be created from scratch anyway, so neither us nor them would gain anything from trying to fit a square into a circle and "reuse" these standards.

Also by trying to make ultra-generic standards you always end up with standards that are bloated and hard to implement, confusing, ambiguous, and in 99% of the cases that is to no benefit of anyone whatsoever. On Nostr we've been trying to make standards that are simple, small, very specific.

the most impactful thing for interoperability would be using content-addressability in a more general way. I.e. having a mapping from hashes (event IDs) to the exact bytes events are serialized to

But this is the case. It is just that the hashing done with Nostr events is different from whatever IPFS is doing, not better or worse. As far as I know, though, IPFS has also a way to do custom hashing, which is one of my criticisms against it: you could end up with the same file having more than one hash on IPFS.

The custom hashing thing they have on IPFS does make sense, because, for example, if you have a document with 40 pages and you'll split that file in 40 pieces and put them in a merkle tree before hashing, it would be nice if each of these 40 pieces corresponded to one page, right? To do that you would need to define a custom hashing scheme for documents with pages of some format etc. You could also define a custom scheme for Nostr events to ensure that the same event has always the same hash and that is not broken by ordering of JSON attributes or spacing -- that's what we did here.

[fiatjaf]

One example of what I was trying to say about the uselessness of reusing these generic standards: you cited that the AT Protocol and Mastodon both "use" webfinger. But what kind of interoperability does that give to them? Can a Mastodon user talk to an AT user because of that? Not at all.

What is happening actually is that AT is using "webfinger-at" and Mastodon is using "webfinger-mastodon", which are two completely independent protocols that just happen to use formats that are similar.

[burdiyan]

Reusing a standard is good if that brings any interoperability benefit, otherwise it is just a waste. Also by trying to make ultra-generic standards you always end up with standards that are bloated and hard to implement, confusing, ambiguous, and in 99% of the cases that is to no benefit of anyone whatsoever.

I could totally agree. There's a balance to this when some standards just don't make much sense to implement. For example, that's why I didn't mention the DID specification. It's just too loose of a standard to be aiming for, IMO. Every DID method feels like its own standard.

For Nostr to work with any other system, a full-fledged custom from ground-up mechanism would have to be created from scratch anyway, so neither us nor them would gain anything from trying to fit a square into a circle and "reuse" these standards.

I'm not sure I agree with this statement. I guess it depends. And my example with having more general content-addressability for Nostr could help the interoperability with IPFS a great deal, without having to do anything custom. You could just put a Nostr event into IPFS if you want, and link to it from other data on IPFS, if the event ID would correspond to the hash of the bytes. Nostr doesn't need to care about IPFS if it doesn't want to, but just by storing data a bit differently it could be much more interoperable.

But this is the case. It is just that the hashing done with Nostr events is different from whatever IPFS is doing, not better or worse.

We might be talking about different ideas here, because I don't see how this (mapping hashes to opaque bytes) is the case in Nostr. Please let me know if I'm missing something.

As far as I understand, Nostr events are serialized, stored and transported as JSON objects. The event ID is derived from the content of the event, but not in the same form it's stored and transported. You have to manipulate the object into some specific form (put fields into JSON array, prepend 0, etc.) in order to validate the hash. And signature is not included in the signing bytes.

So, you can't just say "here's the event ID, and the event bytes, go verify it".

As far as I know, though, IPFS has also a way to do custom hashing, which is one of my criticisms against it: you could end up with the same file having more than one hash on IPFS.

It's true that the same file could be chunked differently and end up with a different hash. But it's just a side effect of laying out bytes differently. But it always works as hash + opaque bytes at every layer. The idea is that you don't need to understand the data in order to verify its ID. So, no matter what piece of data you're given along with its hash, you can always just run the same hash and check it.

So, there's no way to put a Nostr event with its signature into IPFS and end up with the same hash that would correspond to the event ID.

On Nostr we've been trying to make standards that are simple, small, very specific.

I'm buying that the idea of client-relay architecture fits the definition of "simple". That's a great idea to make a system that is simple, small and effective. But for some other things (some of which I mentioned in nostr-protocol/nips#225) I couldn't agree that they are aiming for simplicity. Of course everyone has a different definition of "simple" so it's probably pointless to argue about.

One example of what I was trying to say about the uselessness of reusing these generic standards: you cited that the AT Protocol and Mastodon both "use" webfinger. But what kind of interoperability does that give to them? Can a Mastodon user talk to an AT user because of that? Not at all.

What is happening actually is that AT is using "webfinger-at" and Mastodon is using "webfinger-mastodon", which are two completely independent protocols that just happen to use formats that are similar.

I can't say for sure if AT and Mastodon could interoperate because they "use" common WebFinger spec, but it may be something that is much easier to solve. And rather than talking about "Mastodon user talking to an AT user", I think the benefit here could be that the same username could be shared among systems. So person@example.com could share this username among all the systems by just hosting a single web finger configuration page, saying that "my Mastodon server is this one", "my Nostr relays are these ones", and "my Nostr relays are these ones", all on the same page.

[snarfed]

Minor correction, AT Protocol doesn't use WebFinger. Its identity and discovery are based on DIDs, currently did:plc and soon also did:web. It uses DNS domains as user-visible handles, which are currently verified bidirectionally with a DNS TXT record and a record in the DID doc, similar to NIP-05.

[dentropy]

I believe that Nostr could still be Nostr, if the event ID was derived from the exact bytes of the event, including the signature. This would also make Nostr events perfect IPFS blocks, and would be much easier to implement the validation for anyone (just hash the bytes and verify the hash).

Sounds like this need to be a NIP, the multiformats library is available in Go, Javascirpt, Python, and Rust so it can easily be built into existing clients and relays.

But this is the case. It is just that the hashing done with Nostr events is different from whatever IPFS is doing, not better or worse. As far as I know, though, IPFS has also a way to do custom hashing, which is one of my criticisms against it: you could end up with the same file having more than one hash on IPFS.

If you want custom hashing look no further than than DAG-JSON would have to be used to encode the events in a regular reproducible way.

DAG-JSON can also be used to batch events together.

Reusing a standard is good if that brings any interoperability benefit, otherwise it is just a waste. For Nostr to work with any other system, a full-fledged custom from ground-up mechanism would have to be created from scratch anyway, so neither us nor them would gain anything from trying to fit a square into a circle and "reuse" these standards.

If nostr relays has a eventV2 standard that used CID's (Content Identifiers) nothing would have to change reguarding the core Nostr stack. IPFS already did a transition between CIDv1 and CIDv2 where all CIDv2 ID's can be represented as CIDv1 ID's.

From https://news.ycombinator.com/item?id=37351668
First, I want a replication strategy. Nostr messages get lost in time, and many of the clients end up just blasting an entire message history at your client. Because there's no clue in the protocol how messages are related other than a timestamp this also means you can fake timestamps and write fake messages in the future or back in time. This doesn't have to be an append-only log, but you need some idea of message order to avoid wasting bandwidth to get someone's timeline and detect when a message has been posted out of order.

The DAG-JSON + CAR file formats used in IPLD can help with the data replication stratagy and Nostr Clients just requiresting a firehose of data from relays.