Syncing #62

ErikBjare · 2019-08-28T11:25:03Z

Started working on syncing

… entrypoint for syncing binary

ErikBjare · 2019-09-05T16:43:33Z

@johan-bjareholt Could I get some comments on this?

johan-bjareholt

I need some more time to look into this, still don't fully understand what's going on.

src/sync.rs

johan-bjareholt

A good start.

We need some restructuring later though, such as move this out to a seperate "sync" folder instead of being inside "src" and then moving out everything in "src" into "core" and "server" or something like that. We don't want everything to become a huge monolith. This doesn't need to be done in this PR.

It would be good if you could modularize it a bit more and write it as multiple tests instead and just keep a non-functional main function for now until everything is properly implemented.

I've looked it though a bit and I think I (kind of) understand the flow of the program, I'm not sure if it's simply because I don't fully understand how it's supposed to work or if the functions are oddly split so I have a hard time understanding the overview.

src/datastore/datastore.rs

johan-bjareholt · 2019-09-07T13:44:12Z

src/sync.rs

+        }
+
+        // Sync events
+        // FIXME: Events are not being saved, does the datastore worker need more time before exit?


The events are probably not commited.
You could add a "ForceCommit" command to the datastore if you want.
In addition you could also add an "Stop" command which finishes up and then exits the thread and then the requester thread can wait for the worker thread to exit.

I fixed it by simply moving the references to the datastores out of the main function and then leaving a sleep in main to give it some time to commit (is this necessary though?).

While that kinda fixed it, it still leaves all kinds of unexpected behavior so until we add a ForceCommit or something I've just set commit: true when initializing the DatastoreInstance for now.

Nevermind, added the ForceCommit, was really easy.

Still leads to loads of unexpected behavior though. For example: If I do a insert_events and then a get_event_count I expect it to return the new number, not the old one (there are a lot of cases like this).

The solution as I see it is to commit pending operations before a get operation. (or on every non-heartbeat command)

I would expect that if we ran the aw-server/aw-client integration tests targeting aw-server-rust we'd trigger many of these weird cases.

The solution as I see it is to commit pending operations before a get operation.

This is how the sqlite storage method in python does it, might be a good idea yes.

src/sync.rs

johan-bjareholt · 2019-09-09T14:38:56Z

src/datastore/datastore.rs

                        Ok(Responses::BucketMap(ds.get_buckets()))
                    },
                    Commands::InsertEvents(bucketname, events) => {
+                        ds.commit = true;


I won't until we've fixed the commit-before-get. If I do I have to put ds.force_commit() after practically every ds.insert_events() which is really annoying and hard to debug.

src/datastore/datastore.rs

johan-bjareholt · 2019-09-09T14:46:55Z

src/sync_main.rs

+    // Needed to give the datastores some time to commit before program is shut down.
+    // 100ms isn't actually needed, seemed to work fine with as little as 10ms, but I'd rather give
+    // it some wiggleroom.
+    std::thread::sleep(std::time::Duration::from_millis(100));


In C you don't need to sleep but can wait for a thread to close down.

How to do it properly in Rust without any risk of deadlocks though I have no idea...

If you find a way we could always wait and have a timeout to avoid a deadlock.

johan-bjareholt · 2019-09-09T14:48:07Z

src/sync.rs

+
+/// Returns the sync-destination bucket for a given bucket, creates it if it doesn't exist.
+fn get_or_create_sync_bucket(bucket_from: &Bucket, ds_to: &Datastore) -> Bucket {
+    // Ensure the bucket ID ends in "-synced"


Would be cool with a "synced" property to the buckets instead.

johan-bjareholt

How is this all supposed to work when syncing while aw-server is running?
Will there be writes to the same sqlite file from two processes?

ErikBjare · 2019-09-09T14:54:55Z

@johan-bjareholt The syncing could either happen in the same process, or in a separate process if all write-access to the local aw-server happens through the REST API (which is what I'm considering for the MVP).

johan-bjareholt

After you have fixed the commit before get we can merge this.

The syncing code still needs some improvement, but as long as we don't break aw-server I'm fine with having merged prototype code.

ErikBjare · 2019-09-11T07:25:26Z

@johan-bjareholt If it's not too much effort I'd like it if you fixed the commit-before-get. It shouldn't be too much work and I think you'd do a much better job (not to mention you'd do it a lot quicker).

johan-bjareholt · 2019-09-12T09:33:50Z

@ErikBjare Have you tried to change the transaction behavior to IMMEDIATE instead of EXCLUSIVE?
That might actually solve the problem without having to do commits on every GetEvents as long as we only read one database and are the only ones writing to the other database (there has to be a clear master/slave scenario, which is a good thing anyway).

ErikBjare · 2019-09-16T08:44:56Z

Have you tried to change the transaction behavior to IMMEDIATE instead of EXCLUSIVE?

Nope, the SQLite docs say that "EXCLUSIVE and IMMEDIATE are the same in WAL mode".

Can I merge this so we can get on with merging #65?

johan-bjareholt · 2019-09-16T20:37:43Z

Feel free to merge, I don't approve of the syncing code yet but that's just a prototype. I don't approve of the ds.commit=true either but it's not broken and I can fix that in #65 so it's fine.

You just need to fix some conflict?

ErikBjare and others added 4 commits August 28, 2019 13:24

binary now uses the library (removing duplicate imports), added basic…

58904ba

… entrypoint for syncing binary

fixed tests

7494895

progress on syncing

c93c8a0

datastore: fix blinking test

dee8ee5

improved logging output, added minor test

e09717e

johan-bjareholt reviewed Sep 6, 2019

View reviewed changes

src/sync.rs Outdated Show resolved Hide resolved

johan-bjareholt reviewed Sep 8, 2019

View reviewed changes

ErikBjare added 3 commits September 9, 2019 16:17

major progress on syncing, added tests

7159b77

cleaned up code, added comments

c143fd5

added workaround for low-precision clocks

94cb00d

johan-bjareholt reviewed Sep 9, 2019

View reviewed changes

ErikBjare requested a review from johan-bjareholt September 9, 2019 14:55

johan-bjareholt reviewed Sep 11, 2019

View reviewed changes

moved sync-files into subfolder

dc9d88d

ErikBjare added 3 commits September 16, 2019 10:52

Merge branch 'master' into dev/syncing

926bcd6

renamed commit to force_commit

7c16e89

added sync origin to bucket data

672e20d

fixed type error and got cfg'd out lazy_static on non-Android platforms

6280ada

ErikBjare merged commit 589115f into master Sep 20, 2019

ErikBjare deleted the dev/syncing branch September 20, 2019 10:31

This was referenced Feb 1, 2023

build(deps): bump android_logger from 0.11.0 to 0.12.0 mkvoya/aw-server-rust#23

Closed

build(deps): bump android_logger from 0.11.3 to 0.12.0 #337

Closed

dependabot bot mentioned this pull request Mar 1, 2023

build(deps): bump android_logger from 0.11.0 to 0.13.0 mkvoya/aw-server-rust#26

Open

dependabot bot mentioned this pull request Mar 1, 2023

build(deps): bump android_logger from 0.11.3 to 0.13.0 #346

Closed

SaadBazaz mentioned this pull request Aug 20, 2024

Add webhooks #491

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Syncing #62

Syncing #62

ErikBjare commented Aug 28, 2019

ErikBjare commented Sep 5, 2019

johan-bjareholt left a comment

johan-bjareholt left a comment

johan-bjareholt Sep 7, 2019

ErikBjare Sep 9, 2019

ErikBjare Sep 9, 2019

ErikBjare Sep 9, 2019

ErikBjare Sep 9, 2019 •

edited

Loading

johan-bjareholt Sep 9, 2019

johan-bjareholt Sep 9, 2019

ErikBjare Sep 9, 2019 •

edited

Loading

johan-bjareholt Sep 9, 2019

ErikBjare Sep 9, 2019

johan-bjareholt Sep 9, 2019

johan-bjareholt left a comment

ErikBjare commented Sep 9, 2019

johan-bjareholt left a comment

ErikBjare commented Sep 11, 2019

johan-bjareholt commented Sep 12, 2019

ErikBjare commented Sep 16, 2019 •

edited

Loading

johan-bjareholt commented Sep 16, 2019

Syncing #62

Syncing #62

Conversation

ErikBjare commented Aug 28, 2019

ErikBjare commented Sep 5, 2019

johan-bjareholt left a comment

Choose a reason for hiding this comment

johan-bjareholt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ErikBjare Sep 9, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ErikBjare Sep 9, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johan-bjareholt left a comment

Choose a reason for hiding this comment

ErikBjare commented Sep 9, 2019

johan-bjareholt left a comment

Choose a reason for hiding this comment

ErikBjare commented Sep 11, 2019

johan-bjareholt commented Sep 12, 2019

ErikBjare commented Sep 16, 2019 • edited Loading

johan-bjareholt commented Sep 16, 2019

ErikBjare Sep 9, 2019 •

edited

Loading

ErikBjare Sep 9, 2019 •

edited

Loading

ErikBjare commented Sep 16, 2019 •

edited

Loading