Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
branch: master
Commits on Mar 19, 2012
Commits on Mar 18, 2012
  1. Merge remote-tracking branch 'stumbleupon/master'

    Will Moss authored
    * stumbleupon/master: (36 commits)
      Upgrade to asynchbase 1.2.0.
      Add parameters to change the foreground and background colors.
      Don't throw unnecessary exceptions when parsing timestamps.
      Javadoc fixes.
      Simplify computation of standard deviations.
      Add one more unit test for stddev aggregator.
      Standard deviation aggregator.
      Fix helper functions when using varying UniqueId sizes.
      Licensing adjustment: allow LGPLv2.1+ instead of LGPLv3+.
      Start version 1.1.0.
      Add a bit of README and some NEWS.
      Fix the knob to enable/disable TSD compactions.
      Use semantic versioning.
      Upgrade to GWT 2.4.0.
      Upgrade to Netty 3.2.7.Final.
      Upgrade to ZooKeeper 3.3.4.
      Upgrade to SLF4J 1.6.4.
      Upgrade to logback 1.0.0.
      Upgrade to PowerMock 1.4.10.
      Upgrade to Mockito 1.9.0.
      ...
Commits on Mar 4, 2012
  1. @tsuna

    Upgrade to asynchbase 1.2.0.

    tsuna authored
    Change-Id: Ibb0d17f7eb77809893a2009052454969183fc2cc
Commits on Feb 27, 2012
  1. @tsuna

    Add parameters to change the foreground and background colors.

    tsuna authored
    The query API now supports two more query string parameters,
    `fgcolor' and `bgcolor' to set (respectively) the foreground
    and background colors.  Acceptable values are of the form
    `xRRGGBB' where RGB values are in hexadecimal.  For `bgcolor'
    the special value `transparent' can also be used.
    
    This closes #43 and #94.
    
    Change-Id: I9b451fb1b340fc80d4169a0c8b6c1b37af8f5339
Commits on Feb 23, 2012
  1. @tsuna

    Don't throw unnecessary exceptions when parsing timestamps.

    tsuna authored
    Change-Id: Ib766c8e197f42331f8620c05280757aa10f3d591
  2. @ryang-sfdc @tsuna

    Javadoc fixes.

    ryang-sfdc authored tsuna committed
    Change-Id: Ibb0c006374ba1ca9ee3c65f6fe04a708ad6d2d7f
    Signed-off-by: Benoit Sigoure <tsuna@stumbleupon.com>
Commits on Feb 22, 2012
  1. @tsuna

    Simplify computation of standard deviations.

    tsuna authored
    Change-Id: Icbf7aba4103c029bd858701df82904b21ef0b64c
  2. @tsuna

    Add one more unit test for stddev aggregator.

    tsuna authored
    Change-Id: Icdc510af28e6f6f35b7dd9fc6970d77ff1520a82
  3. @deusaquilus @tsuna

    Standard deviation aggregator.

    deusaquilus authored tsuna committed
    Change-Id: I35aecd7433495fd7bfd99ebb2803416bae5fe0b1
    Signed-off-by: Benoit Sigoure <tsuna@stumbleupon.com>
  4. @tsuna

    Fix helper functions when using varying UniqueId sizes.

    tsuna authored
    By default all UniqueId instances created by OpenTSDB encode the UIDs
    on 3 bytes (for metrics, tag names and tag values).  There were 2 bugs
    that would crop up if one configured tag names and tag values to use
    a different number of bytes to encode the UIDs.
    
    Change-Id: I15a01c513a18e75c69c1ade1abeb3e1602419fa9
Commits on Feb 4, 2012
  1. @tsuna

    Licensing adjustment: allow LGPLv2.1+ instead of LGPLv3+.

    tsuna authored
    Until now, OpenTSDB was available under the terms of the LGPL,
    version 3 or any later version.  This change also allows the
    version 2.1 of the LGPL license.  Those who already accepted
    the terms of the LGPLv3 can keep doing so, because LGPLv2.1
    also states that one can use any later version.
    
    The net result of this change is that one can now use OpenTSDB
    under the terms of LGPLv2.1 or GPLv2, if they so chose.
    
    Change-Id: I67e0d47206db6a5544fdadc9c5fb9cd9466c1ad3
Commits on Dec 23, 2011
  1. @tsuna

    Start version 1.1.0.

    tsuna authored
    Change-Id: If507ed9fedad75da306a24a9abf93d62013c5510
  2. @tsuna

    Add a bit of README and some NEWS.

    tsuna authored
    Change-Id: Ic2bd5f3f1946edbe148a8b15165861a561013f5e
  3. @tsuna

    Fix the knob to enable/disable TSD compactions.

    tsuna authored
    Compactions are enabled if the system property
    `tsd.feature.compactions' is set and its value
    is not the string "false".
    
    Previously the code would enable compactions
    as long as the system property was set, which
    makes it hard to disable them in the future
    once they're enabled by default.
    
    Change-Id: If58a3c269c1b6340c0176fbc793eb9d3fb3dea8e
  4. @tsuna

    Use semantic versioning.

    tsuna authored
    Change-Id: I050ce5a7e6f1919c013d71a8b35dc83a227b4a51
  5. @tsuna

    Upgrade to GWT 2.4.0.

    tsuna authored
    Change-Id: Ib50f47de57523c9ac4125b2eb153b3ee3a6003d1
  6. @tsuna

    Upgrade to Netty 3.2.7.Final.

    tsuna authored
    Change-Id: I1293c3a7d95017e8a4279e0b28d8c4c31bb180fc
  7. @tsuna

    Upgrade to ZooKeeper 3.3.4.

    tsuna authored
    Change-Id: I87141ebbfdc143749644c9bc5d4997e46904b061
  8. @tsuna

    Upgrade to SLF4J 1.6.4.

    tsuna authored
    Change-Id: Ie6acf8c253ab9cad26f86756f5557a7268f2fa20
  9. @tsuna

    Upgrade to logback 1.0.0.

    tsuna authored
    Change-Id: Ifd3d2e81646039e435531047b7a036dd8c19b816
  10. @tsuna

    Upgrade to PowerMock 1.4.10.

    tsuna authored
    Change-Id: Ic49c7e5af251e9a68e4b922b49fed02ffe2198c6
  11. @tsuna

    Upgrade to Mockito 1.9.0.

    tsuna authored
    Change-Id: I422549714e3b464e0dc70ff559db84811c84b92d
  12. @tsuna

    Update to JUnit 4.10.

    tsuna authored
    Change-Id: I46502c17b05a6581f32da9ab8a8b256bc522aa50
  13. @tsuna

    Properly handle malformed query strings.

    tsuna authored
    Change-Id: Iab42fa9429fb32156a22fcf7cdab974a5a2c14db
  14. @tsuna

    Add support for compacted cells in fsck.

    tsuna authored
    Change-Id: Iceeb1ab40026087a3307f24dd38bebf9e21a5198
  15. @tsuna

    Stop (ab)using reflection in tools.

    tsuna authored
    Instead create an `Internal' module that exposes some internal API.
    Although this breaks encapsulation, both the name of the module and
    its Javadoc comment make it clear that this module isn't part of the
    public API.  This makes the code easier to program / maintain than
    with reflection, which is a PITA in Java (and that's not necessarily
    a bad thing).
    
    Change-Id: I6bf27c0b45524f605838db8b4c7b9d721f435a68
  16. @tsuna

    Don't flush the entire compaction queue on shutdown.

    tsuna authored
    It's just too much work and most of it is unnecessary.  Here's why.
    In a typical setup, you'd have at least 2 TSD to be N + 1.  If you
    shutdown the first one, all the traffic will shift to the other one
    by virtue of tcollectors reconnecting elsewhere.  This is going to
    "fault in" almost all the rows that were in the compaction queue of
    the TSD that got shutdown, so the other TSD will eventually compact
    them.  It's possible that few rows will be left behind, and it's OK
    since they can always be compacted later on subsequent reads.
    
    Flushing the entire queue simply takes too long.  If there are, say,
    600k rows sitting in the queue, and the TSD is able to flush 2500/s,
    it would take 4 minutes to flush them all.  So instead we only flush
    the ones we were really supposed to flush anyway, that is the rows
    that are over 1h old.
    
    Change-Id: I4594b714ebf0d831c9fb8d314a38154439166ec0
  17. @tsuna

    Limit the number of concurrent flushes that can happen.

    tsuna authored
    When the CompactionQueue contains, say, 1M rows, and flush() is called,
    the code was attempting to compact all these rows concurrently.  This
    is unrealistic and can either kill HBase and/or run the TSD out of RAM.
    With this change, a single call to flush() can only initiate up to 10k
    concurrent compactions, after which the code asynchronously waits until
    the batch of 10k is done before flushing more.
    
    Change-Id: I05ec3839e08f13455f80b3aba31acf5f42c7ac38
  18. @tsuna

    Make the compaction thread more robust.

    tsuna authored
    A Throwable that isn't an Exception could kill the thread.  This is bad
    as without this thread the TSD will slowly but surely run out of memory.
    We can handle OOME by dropping the queue on the floor to free up memory.
    Other Throwables will just cause the thread to die after 1s, but a new
    one will be created to replace it.
    
    Change-Id: I2bdb65338da43a0f366cdaa6b2b2e7aeabcd503e
  19. @tsuna

    Dynamically compute how many rows to flush per iteration.

    tsuna authored
    This ensures that we flush more aggressively when under pressure.
    
    Change-Id: Ie82f93b080e1841b560e6a34a288cda9e755f65e
  20. @tsuna

    Properly handle throttling requests from asynchbase.

    tsuna authored
    Compactions rewrite so much data that they're very likely to cause
    region splits in HBase.  When they do, they keep sending tons of
    requests to the HBase client, which eventually makes them all fail
    with a PleaseThrottleException, due to the nature of its completely
    non-blocking asynchronous implementation.  When we get such this
    exception, we simply re-queue the row for compaction again, to try
    to compact it later.
    
    Change-Id: Ie7f5c4291c82ce9485243b7265763a399150f3d1
  21. @tsuna

    Add 2 more stats on compaction activities.

    tsuna authored
    Change-Id: Ieb1961516fb96f11257f99d3c1d07efbd325a660
  22. @tsuna

    Make the read path work in O(N) again.

    tsuna authored
    After the changes to `RowSeq' to support reading compacted rows, the
    code kept using naive `DataPointsIterator', which does random access
    into the data structure.  With variable length encoding, we can no
    longer locate the n-th value in O(1), so the code needed a O(N) walk
    to find each data point, leading to an quadratic loop when iterating
    on all the data points in a `RowSeq'.
    
    With this change, `RowSeq' uses its own custom iterator implementation
    instead of `DataPointsIterator' in order to work in O(N) [single pass].
    
    Change-Id: I98d00e1b30463542134e087243c6243e4c03ac3a
  23. @tsuna

    Improved schema: compact the data in HBase.

    tsuna authored
    This is a major change.  New data points written by this code aren't
    readable by older versions of the code.  The new behavior is turned
    off by default, however.  After turning on the new behavior, instead
    of rolling back, you can turn it off again if you observe problems.
    In order to turn on the new compaction code, the JVM system property
    "tsd.feature.compactions" needs to be set.  You can set this by in
    your `tsdb.local' file like so:
      JVMARGS='-Dtsd.feature.compactions'
    
    With this change the TSD remembers every row it's writing to in HBase.
    Periodically, it will read back old rows from HBase, and rewrite them
    in a more efficient format.  The compacted form consists mostly in
    concatenating together all the qualifiers and values inside that row.
    
    When the TSD reads a row that isn't compacted properly (or at all)
    it will fix it by overwriting the row again.  So the code is handling
    properly weird cases such as data points coming in with very old
    timestamps or out of order data etc.  It also handles properly races
    in case multiple concurrent TSDs attempt to compact the same row at
    the same time.
    
    In order for this change to be truly effective, it's important to have
    HBase compactions turned on, this way HBase can purge the individual
    data points and keep the compacted form only.  If you wish to leave
    HBase compactions turned off, consider triggering them periodically
    (either via a cron job, or manually).
    
    Note: `fsck' currently doesn't support the compacted rows, and will
    emit a warning when it sees one.  Another commit will add support for
    compacted rows in `fsck' in the future.
    
    Change-Id: Ibc8f0154f7dd79d5337fa7effb645a297f9f94b2
  24. @tsuna

    Don't log "Connection reset by peer" errors.

    tsuna authored
    They just spam the logs unnecessarily.
    
    Change-Id: I6fb9081474c97142888e85b89e8fe130e0aebcac
Something went wrong with that request. Please try again.