Update nostrdb, switch to local relay model #2121

When trying to build from rust Signed-off-by: William Casarin <jb55@jb55.com>

Closes: damus-io/nostrdb#21 Signed-off-by: William Casarin <jb55@jb55.com>

Right now it's accidently hardcoded. Fixes: 8376e5bca05c ("add "import -"") Signed-off-by: William Casarin <jb55@jb55.com>

everything will be in here soon

rust doesn't like packed structures, so hide this from bindgen This also buttons up the API so less things are exposed which is good. Signed-off-by: William Casarin <jb55@jb55.com>

Signed-off-by: William Casarin <jb55@jb55.com>

This is the start of our rust library for nostrdb. Implement idiomatic interfaces for Ndb and NdbConfig. Changelog-Added: Add initial rust library Signed-off-by: William Casarin <jb55@jb55.com>

otherwise build fails Signed-off-by: William Casarin <jb55@jb55.com>

since it was overwritten when we synced with damus Signed-off-by: William Casarin <jb55@jb55.com>

we will be applying a patch here as well

A lot of this was pulled from core-lightning. Not sure what is actually needed or not. Signed-off-by: William Casarin <jb55@jb55.com>

for now

Signed-off-by: William Casarin <jb55@jb55.com>

since I keep overwriting it by accident Signed-off-by: William Casarin <jb55@jb55.com>

Signed-off-by: William Casarin <jb55@jb55.com>

We will be storing raw nostr bech32 buffers directly into nostrdb, so adapt our bech32 code to reflect this. When doing our content parsing pass, we will only look for strings and we won't allocate any intermediate buffers. Only when we write this string block to nostrdb will we actually allocate in our nostrdb output buffer (no mallocs!) Signed-off-by: William Casarin <jb55@jb55.com>

This is the same as cursor_slice except we don't memset afterwards Signed-off-by: William Casarin <jb55@jb55.com>

This is a varint helper that doesn't pull larger than uint32 Signed-off-by: William Casarin <jb55@jb55.com>

Signed-off-by: William Casarin <jb55@jb55.com>

handy function for padding buffers to some byte alignment Signed-off-by: William Casarin <jb55@jb55.com>

Signed-off-by: William Casarin <jb55@jb55.com>

This adds some initial code for nostrdb content parsing. We still need to write tests for encoding and decoding, so this is likely not working yet. Signed-off-by: William Casarin <jb55@jb55.com>

since we will be decoding these in realtime, let's make sure we can decode them in O(1) Signed-off-by: William Casarin <jb55@jb55.com>

since we modified this recently, let's add some tests to make sure we didn't break anything Signed-off-by: William Casarin <jb55@jb55.com>

We need to pull the data out as well! Let's add some initial decoders. We still need tests to make sure it's working. Signed-off-by: William Casarin <jb55@jb55.com>

Fix parsing URL when encountering a period at the end of the url by setting it as disallowed from being present at the end of a URL. Some characters are disallowed to be present at the end of URLs. Presently, the period character is the only disallowed character. A character is the last character in the URL if it is followed by is_whitespace() or if it's the last character in the string. Signed-off-by: kernelkind <kernelkind@gmail.com> Tested-by: William Casarin <jb55@jb55.com> Signed-off-by: William Casarin <jb55@jb5.com> Signed-off-by: William Casarin <jb55@jb55.com>

This adds an api that walks along and pulls compact note block data out of nostrdb. Signed-off-by: William Casarin <jb55@jb55.com>

Signed-off-by: William Casarin <jb55@jb55.com>

Fix this mistake that we have with ndb_notes where we don't know the total size of the object Signed-off-by: William Casarin <jb55@jb55.com>

In some situations we will need to have owned note blocks. For example, when we try to fetch note blocks from the database and it's not there yet. We will need to parse the content on the spot and return an owned copy, since it will not be immediately available in the database. Add a new flag field to note blocks that lets us know if it's owned by malloc or nostrdb. We the add a free function that checks this flag and frees the object if its set. If it is not set then it doesn nothing because it likely came from the database. Signed-off-by: William Casarin <jb55@jb55.com>

Version 1 to start Signed-off-by: William Casarin <jb55@jb55.com>

When ingesting notes, parse text/longform contents and store them in nostrdb. Signed-off-by: William Casarin <jb55@jb55.com>

so we don't need heap allocation. we will be calling this a lot in tight render loops, we don't want to be allocating on each frame. Signed-off-by: William Casarin <jb55@jb55.com>

Signed-off-by: William Casarin <jb55@jb55.com>

since we have this committed now Signed-off-by: William Casarin <jb55@jb55.com>

This adds some initial code for the nostrdb relay subscription monitor. When new notes are written to the database, they are checked against active subscriptions. If any of the subscriptions are matched, the note primary key is written to the inbox queue for that subscription. We also add an ndb_wait_for_notes() method that simply waits for notes to be written by the subscription monitor. Changelog-Added: Added filter subscriptions Signed-off-by: William Casarin <jb55@jb55.com>

Signed-off-by: William Casarin <jb55@jb55.com>

This makes it a bit more flexible, but maybe we can add quoting in the future that re-enables this. Or maybe a search option Signed-off-by: William Casarin <jb55@jb55.com>

We need to free these resources when we're done with them. Signed-off-by: William Casarin <jb55@jb55.com>

Signed-off-by: William Casarin <jb55@jb55.com>

We missed this in the original subscription code Signed-off-by: William Casarin <jb55@jb55.com>

this is an invalid subscription id Signed-off-by: William Casarin <jb55@jb55.com>

If they are sorted we can do binary search when matching filters like how strfry does it. Signed-off-by: William Casarin <jb55@jb55.com>

This is much more efficient than linear scans Signed-off-by: William Casarin <jb55@jb55.com>

We can just use a list of filters instead when subscribing Signed-off-by: William Casarin <jb55@jb55.com>

Signed-off-by: William Casarin <jb55@jb55.com>

This is useful for positioning LMDB cursors at the start of a query. We will be re-using this in the upcoming query code Signed-off-by: William Casarin <jb55@jb55.com>

Signed-off-by: William Casarin <jb55@jb55.com>

Still a lot more work to do, but this is at least a proof of concept for querying nostrdb using filters. Signed-off-by: William Casarin <jb55@jb55.com>

Signed-off-by: William Casarin <jb55@jb55.com>

Closes: damus-io/nostrdb#25 Signed-off-by: William Casarin <jb55@jb55.com>

I needed this for debugging kind queries Signed-off-by: William Casarin <jb55@jb55.com>

Signed-off-by: William Casarin <jb55@jb55.com>

it's a bit more intuitive Signed-off-by: William Casarin <jb55@jb55.com>

We will probably need a migration for this? Signed-off-by: William Casarin <jb55@jb55.com>

Instead of running queries off filters directly, we do some simple heuristics and determine a reasonable query plan for the given filter. To test this, also add a kind index query plan and add a test for it. We still need tag, author, and created_at index scans. This is up next! Signed-off-by: William Casarin <jb55@jb55.com>

still very early, but works for kinds! Signed-off-by: William Casarin <jb55@jb55.com>

Signed-off-by: William Casarin <jb55@jb55.com>

This was committed to damus, but this should be in nostrdb or else we will lose it when we update. Damus: 84cfeb1 ("nip42: add initial relay auth support") Link: https://groups.google.com/a/damus.io/g/patches/c/Zx3dk01e0yg/m/t59TsVkXAQAJ Signed-off-by: Charlie Fish <contact@charlie.fish> Signed-off-by: William Casarin <jb55@jb55.com>

I don't technically need this but it helps a lot on the swift side of things since I already have code that uses this identifier of a similar structure Signed-off-by: William Casarin <jb55@jb55.com>

swift is kind of dumb when it comes to opaque pointers Signed-off-by: William Casarin <jb55@jb55.com>

Signed-off-by: William Casarin <jb55@jb55.com>

Fixes: 34093cd ("tce: add AUTH to-client-event") Signed-off-by: William Casarin <jb55@jb55.com>

Signed-off-by: William Casarin <jb55@jb55.com>

This allows you to `try pop` multiple items instead of 1 Signed-off-by: William Casarin <jb55@jb55.com>

The polling variant of ndb_wait_for_notes. This makes more sense for realtime apps like notedeck Changelog-Added: Add ndb_poll_for_notes Signed-off-by: William Casarin <jb55@jb55.com>

This fixes a few ownership issues Signed-off-by: William Casarin <jb55@jb55.com>

Signed-off-by: William Casarin <jb55@jb55.com>

This is a pretty scary looking function that realloc our large variable filter buffer into a compact one. This saves up a bunch of memory when we are done building the filter. Signed-off-by: William Casarin <jb55@jb55.com>

Instead of storing exact pointers inside of our filter elements, just store offsets. This will allow us to clone filters very easily without having to mess around with fixing up the pointers afterwards. Signed-off-by: William Casarin <jb55@jb55.com>

Clone filters when moving them into subscriptions. This will allow us to fix the double free issue on the rust side. Signed-off-by: William Casarin <jb55@jb55.com>

Signed-off-by: William Casarin <jb55@jb55.com>

This introduces the basic created_at query plan. We scan the created_at + id index in descending order looking for items that match a filter. This is a very general query plan, but might not be very efficient for anything other than local timelines. Changelog-Added: Add general created_at query plan for timelines Closes: damus-io/nostrdb#26 Signed-off-by: William Casarin <jb55@jb55.com>

This is less efficient for now but we don't have a small-author-list query plan yet. Signed-off-by: William Casarin <jb55@jb55.com>

Signed-off-by: William Casarin <jb55@jb55.com>

A small size_t/uint64 conversion issue Signed-off-by: William Casarin <jb55@jb55.com>

Since Damus iOS is not an immediate-mode UI like android, we would rather not poll for results. Instead we need a way to register a callback function that is called when we get new subscription results. This is also useful on the android side, allowing us to request a new frame to draw when we have new results, instead of drawing every second. Signed-off-by: William Casarin <jb55@jb55.com>

some binding dir stoppers, and configurator

doesn't need to create a copy of the id Signed-off-by: William Casarin <jb55@jb55.com>

This is still kind of broken until queries are switched over to nostrdb. Will do this next Signed-off-by: William Casarin <jb55@jb55.com>

Signed-off-by: William Casarin <jb55@jb55.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update nostrdb, switch to local relay model #2121

Update nostrdb, switch to local relay model #2121

Commits on Apr 25, 2024

Update nostrdb, switch to local relay model #2121

Are you sure you want to change the base?

Update nostrdb, switch to local relay model #2121

Commits on Apr 25, 2024