Permalink
Commits on May 14, 2012
  1. This checkin makes the compression and decompression of sample

    dstryker committed May 14, 2012
    replacable, but turning SampleCoder into an interface.  Lots of files
    got changed because formerly classes that used the coder used static
    methods, and now they need to have access to an instance that
    implements SampleCoder.
    
    I still need to add the right factory class to select alternative
    SampleCoder and TimelineCoder implementations.
    
    util/src/main/java/com/ning/arecibo/util/timeline/samples/SampleCoder.java:
    
        Morph into an interface that only has coding methods - - the
        sample accumulation part got moved to SampleAccumulator, and the
        coding part got moved toe SampleCoderImpl.
    
    util/src/main/java/com/ning/arecibo/util/timeline/samples/SampleAccumulator.java:
    
        New unit containing the portions of the former SampleCoder that
        accumulate samples.
    
    util/src/main/java/com/ning/arecibo/util/timeline/samples/SampleCoderImpl.java:
    
        New unit - - the implementation of new interface SampleCoder.
    
    collector/src/main/java/com/ning/arecibo/collector/guice/CachingDefaultTimelineDAOProvider.java:
    collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineAggregator.java:
    collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineEventHandler.java:
    collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineHostEventAccumulator.java:
    collector/src/main/java/com/ning/arecibo/collector/resources/HostDataResource.java:
    util/src/main/java/com/ning/arecibo/util/timeline/chunks/TimelineChunk.java:
    util/src/main/java/com/ning/arecibo/util/timeline/chunks/TimelineChunkAccumulator.java:
    util/src/main/java/com/ning/arecibo/util/timeline/chunks/TimelineChunkDecoded.java:
    util/src/main/java/com/ning/arecibo/util/timeline/chunks/TimelineChunkMapper.java:
    util/src/main/java/com/ning/arecibo/util/timeline/persistent/DefaultTimelineDAO.java:
    
        Add SampleCoder data member and constructor arg, and pass it to
        other methods and constructors that need it.
    
    collector/src/main/java/com/ning/arecibo/collector/guice/CollectorModule.java:
    
        Bind sampleCoder.
    
    collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineAggregatorDAO.java:
    util/src/main/java/com/ning/arecibo/util/timeline/persistent/TimelineDAOQueries.java:
    
        Remove unneeded @RegisterMapper of TimelineChunkMapper.
    
    collector/src/test/java/com/ning/arecibo/collector/TestTimelineChunkAndTimes.java:
    collector/src/test/java/com/ning/arecibo/collector/TestTimelineHostEventAccumulator.java:
    collector/src/test/java/com/ning/arecibo/collector/persistent/TestDefaultTimelineDAO.java:
    collector/src/test/java/com/ning/arecibo/collector/persistent/TestTimelineAggregator.java:
    collector/src/test/java/com/ning/arecibo/collector/persistent/TestTimelineEventHandler.java:
    collector/src/test/java/com/ning/arecibo/collector/persistent/TimelineLoadGenerator.java:
    collector/src/test/java/com/ning/arecibo/collector/process/TestCollectorEventProcessor.java:
    collector/src/test/java/com/ning/arecibo/collector/process/TestFileBackedBuffer.java:
    collector/src/test/java/com/ning/arecibo/collector/process/TestInMemoryCollectorEventProcessor.java:
    collector/src/test/java/com/ning/arecibo/collector/process/TestSamplesReplayer.java:
    collector/src/test/java/com/ning/arecibo/collector/resources/TestHostDataResource.java:
    util/src/test/java/com/ning/arecibo/util/timeline/chunks/TestTimelineChunkAccumulator.java:
    util/src/test/java/com/ning/arecibo/util/timeline/chunks/TestTimelineChunkToJson.java:
    util/src/test/java/com/ning/arecibo/util/timeline/samples/TestSampleCoder.java:
    util/src/test/java/com/ning/arecibo/util/timeline/samples/TestSampleCompression.java:
    
        Create and use a SampleCoderImpl instance rather than calling
        static methods on the original SampleCoder.
    
    collector/src/test/java/com/ning/arecibo/collector/TestEventCollectorServer.java:
    
        Fix test bug - - the time needs to be rounded to the nearest second.
    
    util/src/main/java/com/ning/arecibo/util/timeline/DecimatingSampleFilter.java:
    
        Reflect the move of getDoubleValue() from SampleCoder to ScalarSample.
    
    util/src/main/java/com/ning/arecibo/util/timeline/samples/ScalarSample.java:
    
        Move methods getDoubleValue() and sameSampleValue() from
        SampleCoder to here, since they are not coder-specific.
Commits on May 11, 2012
  1. This abstracts TimelineCoder and TimeCursor into interfaces, and

    dstryker committed May 11, 2012
    creates TimelineCoderImpl and TimeCursorImpl that implement them.  It
    also changes the API to always use DateTime instances to refer to
    times, since other implementations of TimelineCoder and TimeCursor
    might not use second resolution unix time.
    
    util/src/main/java/com/ning/arecibo/util/timeline/times/TimeCursor.java:
    
        Renamed to TimelineCursor.java
    
    util/src/main/java/com/ning/arecibo/util/timeline/times/TimelineCursor.java:
    util/src/main/java/com/ning/arecibo/util/timeline/times/TimelineCoder.java:
    
        Change former static methods into an interface, and make all
        interface methods use DateTimes rather than ints.
    
    util/src/main/java/com/ning/arecibo/util/timeline/times/TimelineCursorImpl.java:
    util/src/main/java/com/ning/arecibo/util/timeline/times/TimelineCoderImpl.java:
    
        The implementations of the new interfaces, which used to be static
        methods.
    
    collector/src/main/java/com/ning/arecibo/collector/guice/CollectorModule.java:
    
        Bind TimelineCoder.
    
    collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineAggregator.java:
    collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineEventHandler.java:
    collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineHostEventAccumulator.java:
    
        Add TimelineCoder constructor arg and data member, and use it
        rather than calling a static method to combine timelines.
    
    collector/src/test/java/com/ning/arecibo/collector/TestEventCollectorServer.java:
    
        Conform to TimelineCursor API change to use DateTimes.
    
    collector/src/test/java/com/ning/arecibo/collector/TestTimelineChunkAndTimes.java:
    
        Conform to TimelineCoder API change to use DateTimes.
    
    collector/src/test/java/com/ning/arecibo/collector/TestTimelineHostEventAccumulator.java:
    collector/src/test/java/com/ning/arecibo/collector/persistent/TestTimelineAggregator.java:
    collector/src/test/java/com/ning/arecibo/collector/persistent/TestTimelineEventHandler.java:
    collector/src/test/java/com/ning/arecibo/collector/persistent/TimelineLoadGenerator.java:
    collector/src/test/java/com/ning/arecibo/collector/process/TestCollectorEventProcessor.java:
    collector/src/test/java/com/ning/arecibo/collector/process/TestFileBackedBuffer.java:
    collector/src/test/java/com/ning/arecibo/collector/process/TestInMemoryCollectorEventProcessor.java:
    collector/src/test/java/com/ning/arecibo/collector/process/TestSamplesReplayer.java:
    collector/src/test/java/com/ning/arecibo/collector/resources/TestHostDataResource.java:
    util/src/main/java/com/ning/arecibo/util/timeline/chunks/TimelineChunkDecoded.java
    util/src/main/java/com/ning/arecibo/util/timeline/samples/SampleCoder.java
    util/src/main/java/com/ning/arecibo/util/timeline/times/TimeRangeSampleProcessor.java:
    util/src/main/java/com/ning/arecibo/util/timeline/times/TimelineCoder.java:
    util/src/test/java/com/ning/arecibo/util/timeline/chunks/TestTimelineChunkAccumulator.java:
    util/src/test/java/com/ning/arecibo/util/timeline/chunks/TestTimelineChunkToJson.java:
    util/src/test/java/com/ning/arecibo/util/timeline/samples/TestSampleCoder.java:
    util/src/test/java/com/ning/arecibo/util/timeline/samples/TestSampleCompression.java:
    
        Conform to the new TimelineCoder/TimelineCursor APIs
    
    util/src/main/java/com/ning/arecibo/util/timeline/samples/SampleProcessor.java:
    
        Conform to TimelineCursor name change.
    
    util/src/test/java/com/ning/arecibo/util/timeline/times/TestTimelineCoder.java:
    
        Extensive changes to deal with DateTimes rather than int unix
        times in the TimelineCoder and TimelineCursor APIs.
Commits on May 9, 2012
  1. dashboard: handle edge usecase when refreshing the sample kinds tree

    pierre committed May 9, 2012
    There is a usecase where a set of hosts belonging to the same core type
    may expose different sets of beans.
    This patch supports it.
    
    Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
  2. dashboard: don't draw individual beans belonging to a group

    pierre committed May 9, 2012
    Custom groups are bags of beans and associated sample kinds.
    In the sample kinds tree, make sure not to draw beans twice, i.e.
    if one belongs to a group, it won't be drawn at the top-level.
    
    Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
  3. dashboard: draw beans on startup, if necessary

    pierre committed May 9, 2012
    If hosts were previously selected, draw the associated beans in the
    sample kinds tree.
    
    Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
Commits on May 8, 2012
  1. dashboard: UI refactoring, part 2

    pierre committed May 8, 2012
    Integrate the dashboard with the latest /rest/1.0/sample_kinds
    endpoint. Basic functionality works.
    
    Tests still need to be updated.
    
    Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
  2. collector: update /rest/1.0/sample_kinds endpoint

    pierre committed May 8, 2012
    Add the set of hosts each event category maps to.
    
    Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
  3. dashboard: UI refactoring, part 1

    pierre committed May 8, 2012
    Refactored the JS code for hosts and sample kinds tree to use an
    object-oriented (-ish) model. This makes it easier to keep state
    thorough the actions.
    
    Implemented etag mechanism for hosts and added lazy loading for the hosts tree.
    
    This is only part 1 of the refactoring: tests need to be updated and
    the /sample_kinds endpoint changed to add hosts information.
    
    Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
Commits on May 5, 2012
  1. dashboard: start refactoring JS code

    pierre committed May 5, 2012
    Namespace functions, starting with the validations.
    
    Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
Commits on May 4, 2012
  1. dashboard: add sort helper

    pierre committed May 4, 2012
    This will sort an array of objects by a certain key.
    
    Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
  2. dashboard: add the plumbing for legends and groups configuration

    pierre committed May 4, 2012
    Integrate the configuration bits. This changes the
    /rest/1.0/sample_kinds endpoint to return the new
    GroupsAndSampleKindsStore object (cached as needed via the etag).
    and add a new endpoint /rest/1.0/config to return the legends
    configuration.
    
    Also rename SuperGroup to CustomGroup which seems more intuitive.
    
    The associated client updates will be in a separate commit.
    
    Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
Commits on May 3, 2012
  1. dashboard: add GroupsAndSampleKindsStore

    pierre committed May 3, 2012
    The GroupsAndSampleKindsStore is a caching store for all the samples
    categories/kinds pairs, as well as custom groups.
    
    This will replace the current output of the /rest/1.0/sample_kinds
    endpoint, in order to build and cache the whole sample kinds tree
    on the client side.
    
    Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
  2. dashboard: add backend for legends configurations

    pierre committed May 3, 2012
    This is to be able to configure legends (y-axis) marks.
    
    The legendGroupsFile is used to specify legends to use on a per
    kind basis, wheres the legendFixturesFile define Javascript
    functions that generate the marks.
    
    Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
Commits on May 1, 2012
  1. Add a "decimation_mode" query parameter to the endpoints that fetch

    dstryker committed May 1, 2012
    samples.  decimation_mode defaults to "peak_pick", the previous
    behavior.  However, in addition to peak_pick, we now support
    decimiation_mode of "average".  This will only be useful for runs that
    span a considerable time period, and may be obviated by the dashboard
    "smoothing" control.
    
    util/src/main/java/com/ning/arecibo/util/timeline/DecimationMode.java:
    
        New enum defining the two modes.
    
    collector/src/main/java/com/ning/arecibo/collector/resources/HostDataResource.java:
    
        Add query parameter decimation_mode, defaulting to "peak_pick" to
        all the sample-fetching endpoints, and to the methods that create
        decimation filters.
    
    collector/src/test/java/com/ning/arecibo/collector/resources/TestHostDataResource.java:
    util/src/test/java/com/ning/arecibo/util/timeline/TestDecimatingFilter.java:
    
        Add DecimationMode args to calls to DecimatingSampleFilter
        constructor calls.
    
    util/src/main/java/com/ning/arecibo/util/timeline/DecimatingSampleFilter.java:
    
        Add DecimationMode arg to the constructor.
    
        When processing samples, if the decimationMode is PEAK_PICK, do
        what we used to do.  If it's AVERAGE, average the samples.
Commits on Apr 30, 2012
  1. dashboard: add back the direct link to the raw data

    pierre committed Apr 30, 2012
    Add a per-graph link to get the raw data.
    
    Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
  2. dashboard: make sure to have unique ids

    pierre committed Apr 30, 2012
    Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
  3. dashboard: add dates in graphs title

    pierre committed Apr 30, 2012
    Display the from and to dates for each graph, this is especially
    useful when navigating through time via the controls.
    
    Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
  4. This checkin contains no semantic changes; it just breaks the

    dstryker committed Apr 30, 2012
    formerly-monolithic com.ning.arecibo.util.timeline into several new
    packages: .samples, .chunks, and .times.  It also moves the DAO
    machinery into com.ning.arecibo.util.timeline.persistent.
  5. util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunk.java:

    dstryker committed Apr 30, 2012
        Eliminate base class CachedObject, and change objectId to chunkId.
    
    util/src/main/java/com/ning/arecibo/util/timeline/CachedObject.java:
    util/src/main/java/com/ning/arecibo/util/timeline/LRUObjectCache.java:
    util/src/main/java/com/ning/arecibo/util/timeline/TimelineCacheManager.java:
    
        Delete these now-unreferenced units.
    
    collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineAggregator.java:
    util/src/main/java/com/ning/arecibo/util/timeline/TimelineChunkBinder.java:
    
        Use TimelineChunk.getChunkId() rather than getObjectId().
    
        Change the type of hostId and smapleKindId from Integer to int.
    
    util/src/test/java/com/ning/arecibo/util/timeline/TestTimelineChunkToJson.java:
    
        Increment the number of keys expected by 1, because we now get
        chunkId.
    
    util/src/main/java/com/ning/arecibo/util/timeline/TimelineDAO.java:
    collector/src/test/java/com/ning/arecibo/collector/MockTimelineDAO.java:
    util/src/main/java/com/ning/arecibo/util/timeline/CachingTimelineDAO.java:
    util/src/main/java/com/ning/arecibo/util/timeline/DefaultTimelineDAO.java:
    
        Change the return types of getOrAddHostId(),
        getOrAddEventCategory(), and getOrAddSampleKindId() to be int
        rather than Integer.
  6. In streamingAggregateLevel(), set up the statement locator for the qu…

    dstryker committed Apr 30, 2012
    …ery object and get rid of the naked sql in Java code
  7. This checkin changes how chunks are gathered for aggregation. The

    dstryker committed Apr 30, 2012
    previous code ran a query that only fetched chunks if the count by
    hostId and sampleKindId was large enough that we knew we could
    aggregate them.  That query ran very fast in sub-gigabyte dbs, but
    runs slowly in multiple-gigabyte dbs.
    
    This checkin instead fetches all chunks that might be aggregation
    candidates, which means many of them won't have sufficient numbers to
    be aggregated.  But we can reading is comparatively fast - - the
    aggregator can read an aggregation level worth of chunks in something
    like 15 seconds.  It's the writing and deleting that's expensive.
    
    collector/src/main/java/com/ning/arecibo/collector/guice/CollectorConfig.java:
    
        Change the default aggregation interval from 10 minutes to 2
        hours, because there's no point in reading unaggregatable chunks
        many times.
    
    collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineAggregator.java:
    
        Declare PACKAGE and the chunk mapper, and keep the dbi constructor
        arg.
    
        In aggregateTimelineCandidates(), step through the
        timelineChunkCandidates correctly.
    
        Do sleeping and batch counting in performWrites().
    
        Don't expose getAndProcessTimelineAggregationCandidates() as a
        @Managed operation, because it runs forever.  Instead, add
        @Managed operation initiateAggregation(), which spawns a thread
        and returns.
    
        Replace method aggregateLevel() with streamingAggregateLevel(),
        which creates a query iterator that fetches all chunks at the
        given aggregation level, passing them to a TimelineChunkConsumer
        that accumulates chunks for aggregation.
    
        Fix indentation problems.
    
    collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineAggregatorDAO.java:
    
        Remove now-unused method getTimelineAggregationCandidates().
    
    collector/src/main/resources/com/ning/arecibo/collector/persistent/TimelineAggregatorDAO.sql.stg:
    
        Replace getTimelineAggregationCandidates() with
        getStreamingAggregationCandidates().  The latter has no joins.
Commits on Apr 29, 2012
  1. Change the sample interval from 1 minute to 10 seconds to ensure that…

    dstryker committed Apr 29, 2012
    … the accumulator doesn't extract and save the chunk, messing up chunk counts
  2. Start the TimelineHostEventAccumulator random number generator with a…

    dstryker committed Apr 29, 2012
    … known seed, to make tests repeatable
Commits on Apr 28, 2012
  1. collector/src/main/java/com/ning/arecibo/collector/EventCollectorServ…

    dstryker committed Apr 28, 2012
    …er.java:
    
        Provide a jmx operation to stop the core.
    
    collector/src/main/java/com/ning/arecibo/collector/guice/CollectorConfig.java:
    
        Default aggregationSleepBetweenBatches to 50ms.
    
    collector/src/main/java/com/ning/arecibo/collector/persistent/EventReplayingLoadGenerator.java:
    
        Make initiateShutdown() jmx-accessible, and when it's called, shut
        down the replayer as well.
    
    collector/src/main/java/com/ning/arecibo/collector/persistent/TimelineAggregator.java:
    
        Add counters to time db accesses.
    
    util/src/main/java/com/ning/arecibo/util/timeline/persistent/Replayer.java:
    
        Add method initiateShutdown, to tell the replayer to stop.
Commits on Apr 27, 2012
  1. collector/src/main/java/com/ning/arecibo/collector/persistent/Timelin…

    dstryker committed Apr 27, 2012
    …eEventHandler.java:
    
        Add counters for purged host accumulator maps, and purged
        accumulators.
    
    util/src/main/java/com/ning/arecibo/util/timeline/TimelineCoder.java:
    
        Paper over the case that can't happen in real life, but did happen
        when I restarted the load generator: aggregating timelines where
        the first time earlier than the last time of a proceeding
        timeline.
  2. collector/src/main/java/com/ning/arecibo/collector/persistent/EventRe…

    dstryker committed Apr 27, 2012
    …playingLoadGenerator.java:
    
        Add machinery to log time spent sending and time spent sleeping.
    
        Set replayIterationStartTime in a way that guarantees no
        overlapping samples.
    
        Add property to say whether we process samples, and logging
        interval.
  3. dashboard: switch loading indicator to a modal

    pierre committed Apr 27, 2012
    Use a modal that covers the screen while data is loading.
    This should make it more obvious to the user that the
    dashboard is doing something and prevent him from further
    clicking.
    
    Note that nothing prevents him (while the modal is activated)
    to press Esc. and/or click outside of the 'Data is loading...'
    box to dismiss the modal. This won't stop the request in progress
    though.
    
    Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
  4. dashboard: update graph buttons CSS

    pierre committed Apr 27, 2012
    Put some space around the groups of graph control icons.
    
    Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
  5. dashboard: update graph controls CSS

    pierre committed Apr 27, 2012
    Put some space around the label names.
    
    Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
  6. dashboard: default to 500 samples

    pierre committed Apr 27, 2012
    screen.width seems to be too generous. By experience,
    David reports that there is no added value in having
    output_count > 500.
    
    Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
  7. dashboard: switch to the munin palette

    pierre committed Apr 27, 2012
    The colorwheel palette we were using had 8 predefined
    colors. Starting with the 9th host, it would default
    to #808080 (gray).
    
    Switch to the munin palette instead, which has 29 colors
    (including gray). Note that it will still default to gray
    starting with the 30th host.
    
    Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>
  8. dashboard: add tooltips for the graph controls

    pierre committed Apr 27, 2012
    The strings used for the tooltips come from:
    
      https://github.com/shutterstock/rickshaw
      https://github.com/mbostock/d3/wiki/SVG-Shapes
    
    They may need some work - it's hard to describe the functions in
    so few words.
    
    Signed-off-by: Pierre-Alexandre Meyer <pierre@ning.com>