Attempt to harden and make aw-client more resilient by ErikBjare · Pull Request #28 · ActivityWatch/aw-client

ErikBjare · 2018-01-30T11:37:42Z

Switched to using persist-queue instead of home-rolled variant

TODO

Ensure that invalid events aren't added to the queue (causing queue-blockage), as @ahnlabb learned the hard way.
- Done, just did a simple assert isinstance(data, dict) for now (should only happen to people who develop new watchers anyway)
Use persist-queue properly (notably the behavior of task_done())
- Fixed in b05da4a

ErikBjare · 2018-01-30T11:42:06Z

Seems to work great, only one issue:

2018-01-30 12:40:22 [WARNING]: auto_commit=False is still experimental,only use it with care.  (persistqueue.sqlbase:100)

Not sure how experimental it is, perhaps we should reach out to the developer to find out.

ErikBjare · 2018-01-30T12:38:24Z

I should add that I'm not certain this behaves the intended way in some circumstances, namely:

When I do a queue.get(), the request fails, and queue.task_done() is not called, will the same object be returned on the second get()? Probably not.
- Edit: Fixed in b05da4a

There's also an additional issue for future plans (but no deal breaker since it would add significant, hard-to-maintain, complexity to the preexisting solution as well):

If we want to push to the front of the queue or modify the last pushed event (such as heartbeat_merge) that is not supported. Although it might not be that hard to modify the library to do this.

ErikBjare · 2018-01-30T13:51:26Z

I think I found another option: shelve in the standard library.

Edit: shelve could work, but might use a lot of memory and eat a significant chunk of CPU if the queue grows large. The best option still seems to be persist-queue, but it'd take some modifications to work as we want it to.

Edit 2: the cons of shelve might not be that bad if we also implement client-side heartbeat_merge since that makes the queue unlikely to grow very large.

johan-bjareholt · 2018-01-30T20:39:18Z

Not sure how experimental it is, perhaps we should reach out to the developer to find out.

Yeah probably. auto_commit=True is set in all the examples in the README so it does seem encouraged there atleast which is odd.

peter-wangxu · 2018-02-19T09:48:23Z

@ErikBjare I came across that your were using persist-queue, just want to answer some your doubt here:

Yeah probably. auto_commit=True is set in all the examples in the README so it does seem encouraged there atleast which is odd.

The warning has been removed in v0.3.5 now as the auto_commit=False is tested and performing very well.

Thanks
Peter

ErikBjare · 2018-02-19T11:44:02Z

@peter-wangxu Awesome! Thanks for reaching out!

…irst call

ErikBjare · 2018-03-08T13:37:50Z

@johan-bjareholt This is ready for review, please review meticulously. I'll test it on my machine for the coming days.

johan-bjareholt · 2018-03-08T16:37:48Z

Currently testing locally, seems to work. Will have a further look at the code itself later.

Still seems to recover incredibly slowly though same as previously, we really need to add the ability to send batches of heartbeats because sending heartbeats and events syncronously over HTTP is incredibly slow even locally. If i queue up heartbeats for 2min it takes another 2min for it to recover those events.

Maybe that's the issue why it is delayed on windows, because HTTP requests locally are even slower and doesn't have time to get ahead again after a bit of queueing? Just a guess.

ErikBjare · 2018-03-08T19:28:28Z

@johan-bjareholt Yeah, batching or premerging would be two options. I started working on a premerge thingy based on this PR, but it quickly got out of hand. Still worth taking a look at just to see all the crazy edgecases when you need to take things like pulsetime into account (not to mention how it breaks many of the guarantees given by persist-queue): #29

johan-bjareholt

Not done reviewing yet, might be some more.
Seems good though from a first glance.

johan-bjareholt · 2018-03-09T14:55:01Z

        self._registered_buckets = []  # type: List[Bucket]

-        self._queue = queue.Queue()
+        self._attempt_reconnect_interval = 10


Maybe add this to the config?

Maybe, but I think it would be a bit overkill.

johan-bjareholt · 2018-03-09T14:57:50Z

+
+def test_pdb():
+    pdb.set_trace()
+"""


What's this?

pdb is pretty handy when testing. You should try it sometime. Removed it now though.

johan-bjareholt · 2018-03-09T14:59:09Z

+    def add_request(self, endpoint: str, data: dict) -> None:
+        """
+        Add a request to the queue.
+        NOTE: Only supports heartbeats


We used to support both events and heartbeats, what happened to that?

We never really did in a good way, if we would do so properly it would cause a lot of additional complexity.

johan-bjareholt · 2018-03-09T15:02:11Z

-        self.queue_file = os.path.join(queued_dir, self.client.name + ".v{}.json".format(self.VERSION))

-    def _create_buckets(self):
+        persistqueue_path = os.path.join(queued_dir, self.client.name + ".v{}.persistqueue.sql".format(self.VERSION))


.sql usually means that it contains a sql command rather than a sqlite db, should probably be just .db

Good point, will fix.

johan-bjareholt · 2018-03-09T15:04:46Z

    VERSION = 1  # update this whenever the queue-file format changes

-    def __init__(self, client, dispatch_interval=0):
+    def __init__(self, client: ActivityWatchClient, dispatch_interval: float=0) -> None:


Remove default dispatch_interval from class parameter? If it's only for testing aw-client you could just as well just hardcode it.

dispatch_interval was never used in the class, removed.

johan-bjareholt · 2018-03-09T15:07:32Z

            self._create_buckets()
            self.connected = True
-            logger.info("Connection to aw-server established")
+            logger.info("Connection to aw-server established by {}".format(self.client.client_name))


Why was this change useful? Seems unnecessary to have it print the client name here?

In the aw-qt log you couldn't tell which client had connected to the server.

Hm, that sucks. It's a shame that python doesn't print the loggers name in the logmessage by default, it it often very useful.
GStreamer does logging very well with the loggers name, then you can filter them with simple env variables such as GST_DEBUG=mylogger:debug,mylogger2:warn etc.. Very neat.

I'm pretty sure we could set something similar up in aw-core's setup_logging.

ErikBjare · 2018-03-16T22:05:12Z

@johan-bjareholt Fixed a few things you mentioned, merge?

switched to using persist-queue instead of home-rolled variant

0b4af12

ghost assigned ErikBjare Jan 30, 2018

ghost added the review label Jan 30, 2018

ErikBjare mentioned this pull request Jan 30, 2018

Harden aw-client ActivityWatch/activitywatch#170

Closed

ErikBjare added 3 commits February 25, 2018 20:32

started working on tests for RequestQueue

c70aabc

run RequestQueue tests in CI

ca7cc35

bumped version of persist-queue

7411001

ErikBjare mentioned this pull request Feb 26, 2018

AFK and window watcher suddenly stopped logging ActivityWatch/activitywatch#164

Closed

ErikBjare added 3 commits March 8, 2018 14:29

fixed issue with using persist-queue which could lead to lost events

b05da4a

fixed typing and misc readability improvements

d2b1127

fixed bug where queued events wouldn't get returned by _get_next on f…

3dac22f

…irst call

removed unused import

66404c5

johan-bjareholt reviewed Mar 9, 2018

View reviewed changes

ErikBjare added 3 commits March 16, 2018 22:58

changed suffix for persistqueue path

c6f3214

removed unnecessary comment

f475ee9

removed unused parameter

03704b2

johan-bjareholt approved these changes Mar 17, 2018

View reviewed changes

johan-bjareholt merged commit 03c1f14 into master Mar 17, 2018

ghost removed the review label Mar 17, 2018

ErikBjare deleted the dev/harden branch March 18, 2018 12:09

Uh oh!

Conversation

ErikBjare commented Jan 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Uh oh!

ErikBjare commented Jan 30, 2018

Uh oh!

ErikBjare commented Jan 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ErikBjare commented Jan 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

johan-bjareholt commented Jan 30, 2018

Uh oh!

peter-wangxu commented Feb 19, 2018

Uh oh!

ErikBjare commented Feb 19, 2018

Uh oh!

ErikBjare commented Mar 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

johan-bjareholt commented Mar 8, 2018

Uh oh!

ErikBjare commented Mar 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

johan-bjareholt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ErikBjare commented Mar 16, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ErikBjare commented Jan 30, 2018 •

edited

Loading

ErikBjare commented Jan 30, 2018 •

edited

Loading

ErikBjare commented Jan 30, 2018 •

edited

Loading

ErikBjare commented Mar 8, 2018 •

edited

Loading

ErikBjare commented Mar 8, 2018 •

edited

Loading