Merge pull request #2 from sprockets/2.0

Change batch submission logic
sprockets · Nov 9, 2016 · cb705b8 · cb705b8
2 parents 7405312 + deba075
commit cb705b8
Show file tree

Hide file tree

Showing 10 changed files with 204 additions and 110 deletions.
diff --git a/.travis.yml b/.travis.yml
@@ -6,6 +6,7 @@ python:
  - 3.5
 install:
  - pip install -r requires/testing.txt
+ - pip install wheel
 script: nosetests --with-coverage
 after_success:
  - codecov

diff --git a/README.rst b/README.rst
@@ -27,31 +27,35 @@ documentation.
 
 The following table details the environment variable configuration options.
 
-+-------------------------------+--------------------------------------------------+---------------+
-| Variable                      | Definition                                       | Default       |
-+===============================+==================================================+===============+
-| ``INFLUXDB_SCHEME``           | The URL request scheme for making HTTP requests  | ``https``     |
-+-------------------------------+--------------------------------------------------+---------------+
-| ``INFLUXDB_HOST``             | The InfluxDB server hostname                     | ``localhost`` |
-+-------------------------------+--------------------------------------------------+---------------+
-| ``INFLUXDB_PORT``             | The InfluxDB server port                         | ``8086``      |
-+-------------------------------+--------------------------------------------------+---------------+
-| ``INFLUXDB_USER``             | The InfluxDB server username                     |               |
-+-------------------------------+--------------------------------------------------+---------------+
-| ``INFLUXDB_PASSWORD``         | The InfluxDB server password                     |               |
-+-------------------------------+--------------------------------------------------+---------------+
-| ``INFLUXDB_ENABLED``          | Set to ``false`` to disable InfluxDB support     | ``true``      |
-+-------------------------------+--------------------------------------------------+---------------+
-| ``INFLUXDB_INTERVAL``         | How often to submit measurements to InfluxDB in  | ``5000``      |
-|                               | milliseconds.                                    |               |
-+-------------------------------+--------------------------------------------------+---------------+
-| ``INFLUXDB_MAX_BATCH_SIZE``   | Max # of measurements to submit in a batch       | ``5000``      |
-+-------------------------------+--------------------------------------------------+---------------+
-| ``INFLUXDB_MAX_BUFFER_SIZE``  | Limit of measurements in a buffer before new     | ``20000``     |
-|                               | measurements are discarded.                      |               |
-+-------------------------------+--------------------------------------------------+---------------+
-| ``INFLUXDB_TAG_HOSTNAME``     | Include the hostname as a tag in the measurement | ``true``      |
-+-------------------------------+--------------------------------------------------+---------------+
++------------------------------+--------------------------------------------------+---------------+
+| Variable                     | Definition                                       | Default       |
++==============================+==================================================+===============+
+| ``INFLUXDB_SCHEME``          | The URL request scheme for making HTTP requests  | ``https``     |
++------------------------------+--------------------------------------------------+---------------+
+| ``INFLUXDB_HOST``            | The InfluxDB server hostname                     | ``localhost`` |
++------------------------------+--------------------------------------------------+---------------+
+| ``INFLUXDB_PORT``            | The InfluxDB server port                         | ``8086``      |
++------------------------------+--------------------------------------------------+---------------+
+| ``INFLUXDB_USER``            | The InfluxDB server username                     |               |
++------------------------------+--------------------------------------------------+---------------+
+| ``INFLUXDB_PASSWORD``        | The InfluxDB server password                     |               |
++------------------------------+--------------------------------------------------+---------------+
+| ``INFLUXDB_ENABLED``         | Set to ``false`` to disable InfluxDB support     | ``true``      |
++------------------------------+--------------------------------------------------+---------------+
+| ``INFLUXDB_INTERVAL``        | How many milliseconds to wait before submitting  | ``60000``     |
+|                              | measurements when the buffer has fewer than      |               |
+|                              | ``INFLUXDB_TRIGGER_SIZE`` measurements.          |               |
++------------------------------+--------------------------------------------------+---------------+
+| ``INFLUXDB_MAX_BATCH_SIZE``  | Max # of measurements to submit in a batch       | ``10000``     |
++------------------------------+--------------------------------------------------+---------------+
+| ``INFLUXDB_MAX_BUFFER_SIZE`` | Limit of measurements in a buffer before new     | ``25000``     |
+|                              | measurements are discarded.                      |               |
++------------------------------+--------------------------------------------------+---------------+
+| ``INFLUXDB_TRIGGER_SIZE``    | The number of metrics in the buffer to trigger   | ``60000``     |
+|                              | the submission of a batch.                       |               |
++------------------------------+--------------------------------------------------+---------------+
+| ``INFLUXDB_TAG_HOSTNAME``    | Include the hostname as a tag in the measurement | ``true``      |
++------------------------------+--------------------------------------------------+---------------+
 
 Mixin Configuration
 ^^^^^^^^^^^^^^^^^^^

diff --git a/docs/api.rst b/docs/api.rst
@@ -10,6 +10,15 @@ is added to the ``example`` InfluxDB database with the measurement name of
 which calls :meth:`~sprockets_influxdb.shutdown`. :meth:`~sprockets_influxdb.shutdown`
 ensures that all of the buffered metrics are written before the IOLoop is stopped.
 
+Measurements will be sent in batches to InfluxDB when there are
+``INFLUXDB_TRIGGER_SIZE`` measurements in the buffer or after ``INFLUXDB_INTERVAL``
+milliseconds have passed since the last measurement was added,
+which ever occurs first.
+
+The timeout timer for submitting a buffer of < ``INFLUXDB_TRIGGER_SIZE``
+measurements is only started when there isn't an active timer, there is not a
+batch currently being written, and a measurement is added to the buffer.
+
 .. code:: python
 
    import logging
@@ -97,7 +106,8 @@ Configuration Methods
 .. autofunction:: sprockets_influxdb.set_max_batch_size
 .. autofunction:: sprockets_influxdb.set_max_buffer_size
 .. autofunction:: sprockets_influxdb.set_clients
-.. autofunction:: sprockets_influxdb.set_submission_interval
+.. autofunction:: sprockets_influxdb.set_timeout
+.. autofunction:: sprockets_influxdb.set_trigger_size
 
 Request Handler Mixin
 ---------------------

diff --git a/docs/history.rst b/docs/history.rst
@@ -3,6 +3,10 @@
 Release History
 ===============
 
+`2.0.0`_ (09 Nov 2016)
+----------------------
+- Change the way the buffered submission of measurements is handled
+
 `1.4.0`_ (12 Oct 2016)
 ----------------------
 - Make the hostname tag optional
@@ -60,7 +64,9 @@ Release History
 ----------------------
 - Initial release
 
-.. _Next Release: https://github.com/sprockets/sprockets-influxdb/compare/1.3.0...master
+.. _Next Release: https://github.com/sprockets/sprockets-influxdb/compare/2.0.0...master
+.. _2.0.0: https://github.com/sprockets/sprockets-influxdb/compare/1.4.0...2.0.0
+.. _1.4.0: https://github.com/sprockets/sprockets-influxdb/compare/1.3.0...1.4.0
 .. _1.3.0: https://github.com/sprockets/sprockets-influxdb/compare/1.2.0...1.3.0
 .. _1.2.0: https://github.com/sprockets/sprockets-influxdb/compare/1.1.0...1.2.0
 .. _1.1.0: https://github.com/sprockets/sprockets-influxdb/compare/1.0.7...1.1.0

diff --git a/requires/installation.txt b/requires/installation.txt
@@ -1 +1 @@
-tornado>=4.0,<4.3
+tornado>4.0
diff --git a/setup.cfg b/setup.cfg
@@ -11,4 +11,5 @@ exclude = env,build
 cover-branches = 1
 cover-erase = 1
 cover-package = sprockets_influxdb
-verbose = 1
+verbosity = 2
+with-coverage = 1