Initial query docs

architecture.rst

@@ -10,23 +10,16 @@ Known as aw-server, it handles storage and retrieval of all activities/entries i
 
 The server also hosts the Web UI (aw-webui) which does all communication with the server using the REST API.
 
-Clients
--------
-
-The server doesn't do anything very interesting on its own, for that we need clients. Most specifically a certain type of client known as watchers.
-Writing these clients is something we've tried to make as easy as possible by creating client libraries with a clear API.
-
-Currently the primary client library is written in Python (known simply as aw-client) but a client library written in JavaScript is on the way and is expected to have the same level of support in the future.
-
-Client libraries:
-
- - `aw-client <https://github.com/ActivityWatch/aw-client>`_ (Python)
- - `aw-client-js <https://github.com/ActivityWatch/aw-client-js>`_ (JavaScript, work in progress)
+Watchers
+--------
 
 Since aw-server doesn't do any data collection on it's own, we need watchers that observe the world and sent the data off to aw-server for storage.
 
+These utilize the :doc:`aw-client` library for making requests to the aw-server.
+
 For a list of watchers, see :doc:`watchers`.
 
+
 User interfaces
 ---------------
 
@@ -45,13 +38,23 @@ aw-core
 
 The aw-core library contains many of the essential parts of ActivityWatch, notably:
 
- - The `event-model`
+ - The `buckets-and-events`
  - The datastore layer
  - Utilities (configuration, logging, decorators)
 
+aw-client
+^^^^^^^^^
+
+Writing these clients is something we've tried to make as easy as possible by creating client libraries with a clear API.
+A client could both be a watcher which sends data as well as a visualizer which fetches and presents data from the aw-server.
+
+Currently the primary client library is written in Python (known simply as aw-client) but a client library written in JavaScript is on the way and is expected to have the same level of support in the future.
+
+ - `aw-client <https://github.com/ActivityWatch/aw-client>`_ (Python)
+ - `aw-client-js <https://github.com/ActivityWatch/aw-client-js>`_ (JavaScript, work in progress)
+
 aw-analysis
 ^^^^^^^^^^^
 
 There are also plans to create a library called `aw-analysis <https://github.com/ActivityWatch/aw-analysis>`_ to aid in
 different types of analysis and transformation one might want to make using ActivityWatch data.
-

buckets-and-events.rst

@@ -0,0 +1,42 @@
+Buckets and Events
+==================
+
+Buckets
+-----------
+
+.. code-block:: javascript
+
+    bucket = {
+      "id": "aw-watcher-mywatcher_myhostname",
+      "created": "2017-05-16T13:37:00.000000",
+      "name": "A short but descriptive human readable bucketname",
+      "type": "mybuckettype",
+      "client": "aw-watcher-mywatcher",
+      "hostname": "myhostname",
+    }
+
+In ActivityWatch we try and separate each kind of datapoint.
+Normally what is most convenient is to have one bucket per client and host.
+For example if we have a watcher which tracks info about the focused window on your computer we would upon startup create a bucket named 'aw-watcher-window_my-host-name' to fill all of our events with datapoints to.
+
+Each bucket also has a buckettype.
+The buckettype specifies the format of the data in the event, so for example for the bucket of our focused window watcher we could have the buckettype 'currentwindow'.
+The buckettype is just a ordinary string, but it is good for clients who want to visualize the data in ActivityWatch to know what the buckets contain.
+
+
+Events
+-----------
+
+The ActivityWatch event model is pretty simple, here's its representation in JSON:
+
+.. code-block:: javascript
+
+    event = {
+      "timestamp": "2016-04-27T15:23:55Z",  // ISO8601 formatted timestamp
+      "duration": 3.14,                     // Duration in seconds
+      "data": {"key": "value"},  // A JSON object, the schema of this depends on the bucket type
+    }
+
+It should be noted that all timestamps are stored as UTC. Timezone information (UTC offset) is currently discarded.
+
+The content in the "data" field could be any JSON object, but it is recommended that every event in a bucket should follow some format depending on the buckettype so the data is easy to analyze.

changelog.rst

@@ -66,4 +66,4 @@ v0.7.2 (planned)
 
 - New query2 API for querying and transforming data
 - Web UI now has a view for the most-visited domains
-
+- Web UI now has a button for removing buckets

examples/client.py

@@ -57,10 +57,11 @@
     # next time the client starts instead.
     sleep(1)
 
-# Synchronous example, insert a shutdown event
-shutdown_data = {"label": "shutdown"}
-shutdown_event = Event(timestamp=now, data=shutdown_data)
-inserted_event = client.insert_event(bucket_id, shutdown_event)
+# Synchronous example, insert an event
+event_data = {"label": "non-heartbeat event"}
+now = datetime.now(timezone.utc)
+event = Event(timestamp=now, data=event_data)
+inserted_event = client.insert_event(bucket_id, event)
 
 # The event returned from insert_event has been assigned an id by aw-server
 assert inserted_event.id is not None

examples/minimal_client.py

@@ -1,7 +1,6 @@
 #!/usr/bin/env python3
 
-from time import sleep
-from datetime import datetime, timedelta, timezone
+from datetime import datetime, timezone
 
 from aw_core.models import Event
 from aw_client import ActivityWatchClient
@@ -13,6 +12,7 @@
 client.create_bucket(bucket_id, event_type="test")
 
 shutdown_data = {"label": "some interesting data"}
+now = datetime.now(timezone.utc)
 shutdown_event = Event(timestamp=now, data=shutdown_data)
 inserted_event = client.insert_event(bucket_id, shutdown_event)
 

examples/query_client.py

@@ -0,0 +1,55 @@
+#!/usr/bin/env python3
+
+from time import sleep
+from datetime import datetime, timedelta, timezone
+
+from aw_core.models import Event
+from aw_client import ActivityWatchClient
+
+client = ActivityWatchClient("test-client", testing=True)
+
+now = datetime.now(timezone.utc)
+start = now
+
+query = "RETURN=0;"
+res = client.query(query, "1970-01-01", "2100-01-01")
+print(res) # Should print 0
+
+bucket_id = "{}_{}".format("test-client-bucket", client.hostname)
+event_type = "dummydata"
+client.create_bucket(bucket_id, event_type="test")
+
+def insert_events(label: str, count: int):
+    global now
+    events = []
+    for i in range(count):
+        e = Event(timestamp=now,
+                   duration=timedelta(seconds=1),
+                   data={"label": label})
+        events.append(e)
+        now = now + timedelta(seconds=1)
+    client.insert_events(bucket_id, events)
+
+insert_events("a", 5)
+
+query = "RETURN = query_bucket('{}');".format(bucket_id)
+
+res = client.query(query, "1970", "2100")
+print(res) # Should print the last 5 events
+
+res = client.query(query, start + timedelta(seconds=1), now - timedelta(seconds=2))
+print(res) # Should print three events
+
+insert_events("b", 10)
+
+query = """
+events = query_bucket('{}');
+merged_events = merge_events_by_keys(events, 'label');
+RETURN=merged_events;
+""".format(bucket_id)
+res = client.query(query, "1970", "2100")
+# Should print two merged events
+# Event "a" with a duration of 5s and event "b" with a duration of 10s
+print(res)
+
+client.delete_bucket(bucket_id)

extending.rst

@@ -9,25 +9,19 @@ We've tried to make things easy for you (and ourselves) so here's some advice on
 Collecting more data
 --------------------
 
-To do this you'd want to write a so-called watcher. Watchers are small programs that collect data and send it off to the server.
+ActivityWatch is written to be flexible to be able to gather most types of data.
+Except for the included aw-watcher-window and aw-watcher-afk which tracks your application usage, there are additional so-called :doc:`watchers` for activitywatch.
+Watchers are small programs that collect data and send it off to the server.
+The only requirement for what kind of data is sent to aw-server as an event is that it has to contain a starttime (and preferably a duration aswell) so it can fit on a timeline.
 
-See :doc:`writing-watchers`.
-
-
-Exporting data programatically
-------------------------------
-
-Exporting data is as simple as:
-
-TODO
-
-
-Writing visualizers
--------------------
-
-Right now, the best way to do this is to simply export the data and transform it yourself to fit the data format you need for your visualization.
-In the future however, we will provide a Python library for common transforms to make things easier for you.
+If you want to write a watcher of your own, see :doc:`writing-watchers`.
 
 
+Fetching Data
+-------------
 
+If you want to fetch data from aw-server for visualization, exporting, backup or something we have not yet thought of, there are a few ways you can do this:
 
+* `Exporting a Bucket <features/exporting-data.html>` If you want a complete dump of all events of bucket
+* `Bucket REST API <./rest.html#get-events>`_ If you want to export raw events in a specific time interval from a bucket
+* `Writing a Query <./querying-data.html#writing-a-query>`_ If you want to summarize/aggregate one or more buckets into more easily readable data

faq.rst

@@ -29,7 +29,7 @@ See the documentation for `extending` or checkout the aw-client repository.
 How do I understand the data that is stored?
 --------------------------------------------
 
-All ActivityWatch data is represented using the `event-model`.
+All ActivityWatch data is represented using `buckets-and-events`.
 
 All events from have the fields :code:`timestamp` (ISO 8601 formatted), :code:`duration` (in seconds), and :code:`data` (a JSON object).
 

features.rst

@@ -5,5 +5,6 @@ Here we will document a few features.
 
 .. toctree::
    features/user-interface
+   features/exporting-data
    features/pausing-logging
    features/filtering-data

features/exporting-data.rst

@@ -0,0 +1,6 @@
+Exporting data
+==============
+
+If you go to the "Raw Data" page in the ActivityWatch webui you can download any of the buckets which contain every collected datapoint in ActivityWatch as a single file.
+
+You can also export data programatically using the REST API, but we do not have a guide for that yet **todo: explain how**

features/user-interface.rst

@@ -9,6 +9,8 @@ ActivityWatch comes with a web interface which currently has the following featu
  - Activity overview
     - Most used applications by day
     - Timeline
+    - Most time spent on a website
+      *(requires the ActivityWatch browser extension)*
  - Bucket overview
     - When a bucket was last updated
     - Listing of the latest events

index.rst

@@ -18,20 +18,21 @@ Welcome to the ActivityWatch documentation!
    faq
    features
    watchers
-   extending
 
 
 .. _dev-docs:
 .. toctree::
    :maxdepth: 3
    :caption: Developer documentation
 
+   extending
    development
    architecture
-   event-model
+   buckets-and-events
+   writing-watchers
+   querying-data
    installing-from-source
    api-reference
-   writing-watchers
    rest
    changelog
 

querying-data.rst

@@ -0,0 +1,65 @@
+Querying Data
+=============
+
+There are a couple of ways to query data in activitywatch.
+
+aw-server supplies an "/query" endpoint (also accesible via aw-client's query method) which supplies a basic scripting language which you can utilize to do transformations on the server-side.
+This option is good for basic analysis and for lightweight clients (such as aw-webui).
+
+Another option is to fetch events from the "/buckets/bucketname/events" endpoint (also accesible via aw-client's get_events method) and either program your own transformations or use transformation methods available in the aw-analysis python library (which includes all transformations available in the query endpoint). This require a lot of more work since you will likely have to reprogram transformations already available in the query API, but on the other hand it is much more flexible.
+
+
+Writing a Query
+---------------
+
+.. note::
+    This section is still WIP.
+    There is still no documentation of all the transform functions, but for most simple queries these examples should be enough.
+
+Queries are the easiest yet advanced way to get events from aw-server buckets in a format which fits most needs.
+Queries can be done by doing a POST request to aw-server either manually or with the aw-client library.
+
+In a query you start by getting events from a bucket and assign that collection of events to a variable, then there are multiple transform functions which you can use to for example filter, limit, sort, and merge events from a bucket.
+After that you assign what you want to receive from the request to the RETURN variable.
+
+Minimal query which only gets events from a bucket and returns it
+
+.. code-block:: bash
+
+    events = query_bucket("my_bucket");
+    RETURN = events;
+
+
+A query which merges events from a bucket in a key1->key2 hierarchy
+
+.. code-block:: bash
+
+    events = query_bucket("my_bucket");
+    events = merge_events_by_keys(events, "merged_key1", "merged_key2");
+    RETURN = events;
+
+
+A simplified query example of how to summarize what programs used while not afk.
+The query intersects the not-afk events from the afk bucket with the events from the window bucket, merges keys from the result and sorts by duration.
+
+.. code-block:: bash
+
+    window_events = query_bucket("window_bucket");
+    not_afk_events = query_bucket("afk_bucket");
+    not_afk_events = filter_keyvals(not_afk_events, "status", "not-afk");
+    window_events = filter_period_intersect(window_events, afk_events);
+    events = merge_events_by_keys(window_events, "appname");
+    events = sort_by_duration(events);
+    RETURN = events;
+
+This is an example of how you can do analysis and aggregation with the query method in python with aw-client
+
+.. literalinclude:: examples/query_client.py
+
+
+Fetching Raw Events
+-------------------
+
+**TODO:** Write this section
+
+`Bucket REST API <./rest.html#get-events>`_