LXMF 0.9.0 Preview - Propagation Stamps & Much Better PN Sync

[markqvist]

I'll have a full post out with a detailed explanation of everything soon, but for anyone who wants to play around with this already, or help test it, here's a pre-release build of LXMF 0.9.0, along with a nomadnet and Sideband build (Android APK and whl package, I can compile a Windows .exe if anyone wants it), that supports the new LXMF functionality.

This release will significantly revamp the propagation node system. Over the last couple of months, there's been a pretty staggering increase in activity on the LXMF side of things. Unfortunately, there's also been a huge influx in what suspiciously looks very spammy, or possibly just people trying to see how much the network can be abused before it breaks. From the stats of my own nodes, this looks like single endpoints delivering a couple of thousand messages all in a row, often to an address that has never announced on the network before. Whatever the purpose has been, it has resulted in the nodes I have access to often handling between 50,000 up to 1,000,000 messages a day.

While this has been a fun and informative load-test of the capacity of LXMF running on relatively limited hardware, it is not really constructive in the long term.

The next release of LXMF activates propagation stamps, meaning that a small (or configurably very large) computational stamp cost must be "paid" for delivering messages to the propagation network. The stamps are very light to validate for nodes and endpoints, but can be dynamically configured to be anywhere from pretty easy (sub-second) to taking several minutes, or even hours to compute.

Before I just start rambling out all the details, I'll leave it relatively short at this. I'll write a proper and structured article soon, but for now, feel free to play around with it and ask any questions you might have.

[markqvist]

Forgot to say that the PN sync scheduling and convergence is also much faster and more efficient now, which is pretty nice given the increased traffic.

The new way propagation nodes signal to each other and sync also opens the way for a lot of functionality in the future, such as sharding, small "personal" nodes, private sub-networks, and much, much more.

[LinuxinaBit]

I saw you added support for Weave which is cool (though I still don't fully understand it 😅), and I know a lot of people are gonna be happy about the Heltec v4 support.

Any thoughts on my LXMF ideas, #837 and #968 ?

Also if you feel up for the read, I have placed my disjointed license and CLA thoughts over the course of your absence here: #850 (it contains replies from other people too)
If you read nothing else, I believe the most important one is https://github.com/markqvist/Reticulum/issues/850#issuecomment-3342118441

[markqvist]

Yeah, it's nice that it's in there now, but the Weave interface module is currently just the driver for connecting to and using Weave devices as Reticulum interfaces. It will need the firmware running on a device to actually be useful ;) The firmware is bascially working currently, and performs quite well, but there's still lots of features and work missing, so it'll be a while before it's available to play around with. Pretty cool to run LXST two-way voice calls over several kilometers using just a couple of ESP32s, though.

Did see all of your other comments, I'll return to that when I have time.

Let's keep this thread focused on the new LXMF 0.9.0 features and propagation stamps, so I'll answer your other LXMF stuff in those threads.

[markqvist]

To clarify some common points of confusion, I'll paste in some comments I made in another thread here as well:

---

There's some common misconceptions regarding how LXMF works, which is understandable, since the only way to know the details of the protocol right now is to read the code and figure it out yourself ;)

LXMF traffic does not use flood routing, neither for direct messages nor propagation nodes. That would imply that everything is broadcast multiple times, which would indeed be very inefficient and stupid.

Instead, the LXMF propagation node sync protocol is designed so that every node that wants a particular kind of message actually gets it from someone in a way that tends toward the highest amount of efficiency and lowest latency. Messages themselves are never transmitted to a node more than once, and before messages are transferred between nodes, they run a precursory negotiation to establish which messages need to be transferred.

Currently, the possible configuration space of what determines which messages a node actually wants is a bit shallow though, and it was always the intention to expand this to a more complex system that would allow automatic network stratification and load-sharing once the core protocol had its kinks worked out - a point we are now reaching.

The 0.9.0 release of LXMF adds the groundwork for this, with a more dynamic peer signalling system embedded in destination announces, specifically from LXMF.py:

# Optional propagation node metadata fields. These
# fields may be highly unstable in allocation and
# availability until the version 1.0.0 release, so use
# at your own risk until then, and expect changes!
PN_META_VERSION        = 0x00
PN_META_NAME           = 0x01
PN_META_SYNC_STRATUM   = 0x02  # <-- Relevant target strata structure for node
PN_META_SYNC_THROTTLE  = 0x03
PN_META_AUTH_BAND      = 0x04
PN_META_UTIL_PRESSURE  = 0x05
PN_META_CUSTOM         = 0xFF

Sometimes, the idea of naively embedding a preferred PN in LXMF delivery endpoint announces is brought up, but this would run counter to the idea of LXMF and Reticulum itself, since it would:

• Have pretty big privacy issues, if everyone had to broadcast to the entire network where they want their messages to end up
• Become an easy attack vector. If you want to mess with a user, it's now very easy to single out a DDoS target.
• Centralize the currently distributed nature of LXMF, and introduce single points of failure to message delivery. The whole point is to have an automated, distributed message store, where people don't have to "pick" a single point of failure like in e-mail, Internet-based instant messaging, and other similar system. You want to be able to download your messages from a large number of nodes, so that if one or many go down, you can still get messages sent to you.
• Require manual hand-holding and configuration for all users. Ideally, we want a system where most users can just fire up an LXMF client, and it will automatically find suitable nodes to send and receive messages through, and simply switch these out on demand if the network topology changes, nodes become unavailable, or the user moves to a completely different network segment.

LXMF aims to have as many messages stored and ready for download as possible within the resource limits specified by the node operator. As I've discussed before, this is the true "concept of Reticulum" - to use allocated resources as intelligently and efficiently as possible, and when resources are exhausted to make intelligent decisions on how to deal with such situations; automatically as long as possible, but when necessary with human intervention.

Ideally, any given area of a physical network should have all relevant LXMF messages available in a distributed store, with some level of redundancy. That does not mean that every node has to store every message, but we want a situation where at least every local network area has as good an approximation towards that as physically possible within resource constraints.

All of this is entirely possible within the limits of current technology, even on relatively modest hardware.

---

When sending messages to a device, they will always be delivered directly by default, and this of course happens directly over Reticulum's routing, using the most efficient path currently available (or, Reticulum's best guess as to what that is - we can't know the best path, in absolute certainty of course, but in practice it is pretty close).

If a device you want to send to is not available on the network for direct delivery, LXMF can try to deliver it via the propagation nodes instead. In this case, the following happens:

• Your device encrypts the message with the destinations ratchet keys. Only the final destination hash of the message is visible, and no information about where the message was sent from, or anything else for that matter, is revealed in the data.
• Your device computes a propagation stamp and attaches it to the message (this makes spamming the network very costly, but is negligible for normal use)
• Your device establishes a direct and efficient connection with a propagation node. This could be a preferred node you have chosen yourself, or a node selected automatically based on network topology or other parameters.
• The connection to the PN is established anonymously, and does not reveal your identity or address. The PN has no idea where the message is coming from, or if the device delivering it is even the originator, or simply passing it on from someone else.
• Your device hands the message over to the PN, and the PN verifies if the propagation stamp is correct. If the stamp is correct, the message is scheduled for distribution queue mapping.
• The PN now determines which other nodes it is peered with would want to have a copy of this message, and adds it do distribution queues for these nodes.
• The PN periodically connects with other nodes, offering them messages they might want, and only transfer the messages if they actually want them
• No messages are ever transferred to a given propagation node more than once.
• Distribution queues between nodes are prioritized according to a weighting algorithm, ensuring that large messages don't hog up the network.
• After some time, messages expire and are removed from the message store of propagation nodes.
• When messages are downloaded by a client, they are removed from the message store (full distributed signalling of this across the entire PN network is not implemented yet, since it depends on another primitive related to sync stratification, but it is coming soon).
• If the resource limits of a node is reached, it cleans out messages according to a weighting algorithm, and tries to keep as many relevant messages as possible, while staying within the defined resource limits.

Now, the main problem that had occurred over the past few months was that while stamps were required to message users directly, making it hard to spam individual users, it was essentially "free" to deliver invalid messages to the propagation network.

Of course, you couldn't actually get unstamped or invalidly stamped messages to be accepted by a destination, even if they were delivered over the propagation network, but what you could do, was just upload massive amounts of random or invalid messages to the network, which would waste a lot of resources for all the involved PNs. Some of my nodes were processing millions of messages a week, for "users" that probably didn't even exist, since the network never saw any announces for them.

With LXMF 0.9.0, this has now been solved, since it would be exceedingly expensive (computationally) to send that many messages to the propagation network, while it is still essentially free (computationally) to validate and process them.