Compute changed keys in pull and mutate #341

arv · 2021-04-22T18:58:49Z

Return the changed keys in mutation commits and maybe end pull.

We now return the touched keys in subscribe queries. We also returned the changed keys in mutation commits and maybe end pull. Towards rocicorp/replicache#63

arv

I think this needs a few rounds of polish review but I think it is ready to start looking at

src/embed/connection.rs

src/db/read.rs

arv · 2021-04-22T19:03:19Z

src/prolly/map.rs

+    }
+
+    /// Returns the keys that are different between two maps.
+    pub fn diff<'a>(a: &'a Self, b: &'a Self) -> Result<Vec<String>, FromUtf8Error> {


One optimization I didn't do is to create key ranges of this... we can do that later.

aboodman · 2021-04-22T19:30:12Z

🎉

One question right off:

We now return the touched keys in subscribe queries.

It seems like the JS would already have the information it needs here? It know what keys are being requested and the scan parameters.

arv · 2021-04-22T21:12:18Z

It seems like the JS would already have the information it needs here? It know what keys are being requested and the scan parameters.

That is true. My reason for moving this to Rust was code reuse in future non JS SDK but maybe that was a bad call? At one point I even wanted to keep track of the subscriptions in Rust but due to multiple tabs I moved the storage up to js.

aboodman · 2021-04-22T21:15:05Z

On Thu, Apr 22, 2021 at 11:12 AM Erik Arvidsson ***@***.***> wrote: It seems like the JS would already have the information it needs here? It know what keys are being requested and the scan parameters. That is true. My reason for moving this to Rust was code reuse in future non JS SDK but maybe that was a bad call? At one point I even wanted to keep track of the subscriptions in Rust but due to multiple tabs I moved the storage up to js.

It's up to you. If you find it easy enough to work in Rust, I don't care. I just don't want hypothetical desire to work on other platforms in the future to slow us down at all now. If you find yourself fighting with rust, and it's easy to move it out to JS, do it.

…

— You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#341 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAATUBFXCOVDYQPTKO4G4FTTKCGMLANCNFSM43NDQ25A> .

arv · 2021-04-22T21:25:50Z

It's up to you. If you find it easy enough to work in Rust, I don't care. I
just don't want hypothetical desire to work on other platforms in the
future to slow us down at all now. If you find yourself fighting with rust,
and it's easy to move it out to JS, do it.

Doing it in JS to see how much complexity it removes.

aboodman · 2021-04-22T21:22:28Z

src/prolly/map.rs

+                Some(pending_val) => match self.base_get(key) {
+                    Some(base_val) => {
+                        if pending_val != base_val {
+                            keys.push(String::from_utf8(key.clone())?);


Why do the conversion to utf8 here?

prolly::Map keys are [u8] so I would expect its diff API to be expressed in those terms too.

Sad to copy/allocate all the strings. Did you consider attempt returning references to the data already in the map? Not sure if it's worth the complexity...

Needs to be copied later so I copied it earlier since it made the code simpler. I can think a bit more about this to see if it makes sense to go [u8] all the way to JS

Going to do this in another PR

src/prolly/map.rs

aboodman · 2021-04-22T21:26:31Z

src/db/read.rs

+    }
+}
+
+pub type Subscription = Arc<RwLock<SubscriptionInner>>;


Do we actually need concurrent mutability of this data? I'm very concerned about the proliferation of locks in our code as it will made deadlock increasingly difficult to avoid.

In theory we do not need locks because JS and Wasm is single threaded but I don't think Rust would allow that.

The following case needs "concurrent" access.

rep.subscribe(tx => Promise.all([tx.get('a'), tx.get('b')]) );

I do not see how this could deadlock but it is always a good thing to keep in mind.

At least some of the proliferation of locks in the rust code is due to rust not allowing inner mutability without them. Where in another language we wouldn't bother putting a lock around a thing that we know is not going to be accessed concurrently, rust makes us.

src/db/read.rs

src/embed/connection.rs

src/embed/types.rs

src/prolly/map.rs

src/sync/pull.rs

arv · 2021-04-27T19:11:25Z

src/sync/pull.rs

@@ -1203,6 +1297,7 @@ mod tests {
                intervening_sync: false,
                exp_replay_ids: vec![],
                exp_err: None,
+                exp_changed_keys: ChangedKeysMap::new(),


@phritz It seems like this test are all about nop mutations and pulls? I was looking for a rust/wasm test that does pull with changes but I guess we didn't need to test this before?

src/sync/pull.rs

arv

Ready for realz. PTAL

arv · 2021-04-29T05:32:45Z

src/prolly/map.rs

+                Some(pending_val) => match self.base_get(key) {
+                    Some(base_val) => {
+                        if pending_val != base_val {
+                            keys.push(String::from_utf8(key.clone())?);


Going to do this in another PR

phritz

I gave this a careful reading and it looks great! Only a couple minor comments.

src/embed/types.rs

src/sync/pull.rs

src/sync/types.rs

Followup to #341

Minimize subscription refire

077ced7

We now return the touched keys in subscribe queries. We also returned the changed keys in mutation commits and maybe end pull. Towards rocicorp/replicache#63

arv requested review from phritz and aboodman April 22, 2021 18:58

arv mentioned this pull request Apr 22, 2021

Only call subscriptions when keys change rocicorp/replicache#365

Merged

arv commented Apr 22, 2021

View reviewed changes

src/embed/connection.rs Outdated Show resolved Hide resolved

src/db/read.rs Outdated Show resolved Hide resolved

arv commented Apr 22, 2021

View reviewed changes

Move subscriptions recording to JS

cd6b7c8

arv changed the title ~~Minimize subscription refire~~ Compute diff pulling and mutating Apr 22, 2021

arv changed the title ~~Compute diff pulling and mutating~~ Compute diff in pull and mutate Apr 22, 2021

aboodman reviewed Apr 25, 2021

View reviewed changes

arv added 3 commits April 25, 2021 12:29

Split commit into two different functions to reducude diff size

974276c

Renamed a few things and some feedback changes

f209dfc

Add changed kesy for indexes in pull

e90000a

arv changed the title ~~Compute diff in pull and mutate~~ Compute changed keys in pull and mutate Apr 27, 2021

arv commented Apr 27, 2021

View reviewed changes

Index pull test... grrr

cd30a7f

arv commented Apr 28, 2021

View reviewed changes

src/sync/pull.rs Show resolved Hide resolved

arv added 3 commits April 28, 2021 15:35

Fixed tests

52b15fb

cleanup?

0d33bb8

Figured out last test

a8417d4

arv commented Apr 29, 2021

View reviewed changes

arv requested a review from aboodman April 29, 2021 18:58

arv merged commit 0e54241 into rocicorp:main Apr 29, 2021

arv deleted the subscribe-track branch April 29, 2021 20:21

phritz reviewed May 3, 2021

View reviewed changes

src/embed/types.rs Show resolved Hide resolved

src/sync/pull.rs Show resolved Hide resolved

src/sync/pull.rs Show resolved Hide resolved

src/sync/types.rs Show resolved Hide resolved

arv added a commit that referenced this pull request May 6, 2021

Comment changes

b8e4426

Followup to #341

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compute changed keys in pull and mutate #341

Compute changed keys in pull and mutate #341

arv commented Apr 22, 2021 •

edited

Loading

arv left a comment

arv Apr 22, 2021

aboodman commented Apr 22, 2021

arv commented Apr 22, 2021

aboodman commented Apr 22, 2021 via email

arv commented Apr 22, 2021

aboodman Apr 22, 2021

aboodman Apr 22, 2021

arv Apr 25, 2021

arv Apr 29, 2021

aboodman Apr 22, 2021

arv Apr 25, 2021

phritz May 3, 2021

arv Apr 27, 2021

arv left a comment

arv Apr 29, 2021

phritz left a comment

Compute changed keys in pull and mutate #341

Compute changed keys in pull and mutate #341

Conversation

arv commented Apr 22, 2021 • edited Loading

arv left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aboodman commented Apr 22, 2021

arv commented Apr 22, 2021

aboodman commented Apr 22, 2021 via email

arv commented Apr 22, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arv left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

phritz left a comment

Choose a reason for hiding this comment

arv commented Apr 22, 2021 •

edited

Loading