Skip to content
This repository has been archived by the owner on Apr 22, 2022. It is now read-only.

Allow for multiple endpoints and many-to-many mappings of endpoints to sinks #115

Merged
merged 103 commits into from
Aug 9, 2016

Commits on Nov 5, 2015

  1. Start of work to support many-to-many sources/sinks/mappings.

    Changes include:
     - A new configuration layout for supporting multiple sources, sinks and mappings.
     - No more enqueuing delay. Queues tend to either be full or empty, and are really only effective at smoothing jitter. Spinning to delay doesn't really help, so we just drop the events immediately instead of blocking.
     - No more session-binning strategy for HDFS. In practice we didn't use this, and it's complicated to do properly when multiple sources (with their own idea of sessions) might be feeding into a sink.
    
    For now, the code only supports the existing single browser-based source and HDFS/Kafka sinks. Configuration validation is also not yet implemented.
    asnare committed Nov 5, 2015
    Configuration menu
    Copy the full SHA
    453e6d5 View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2015

  1. Configuration menu
    Copy the full SHA
    f8ad26c View commit details
    Browse the repository at this point in the history

Commits on Nov 23, 2015

  1. Configuration menu
    Copy the full SHA
    0acf80e View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2015

  1. Configuration menu
    Copy the full SHA
    d32f4ea View commit details
    Browse the repository at this point in the history
  2. Do not throw RuntimeException on MappingException, but instead catch …

    …mapping exceptions and turn them into useful error messages and leave configuratio invalid.
    friso committed Nov 24, 2015
    Configuration menu
    Copy the full SHA
    e51eabb View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    c00ed4e View commit details
    Browse the repository at this point in the history

Commits on Dec 4, 2015

  1. Configuration menu
    Copy the full SHA
    04d9b2b View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2015

  1. Reordering of methods.

    friso committed Dec 7, 2015
    Configuration menu
    Copy the full SHA
    7fa5f73 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    4ffeff7 View commit details
    Browse the repository at this point in the history

Commits on Dec 22, 2015

  1. Merge branch 'master' into dev/many-to-many

    # Conflicts:
    #	build.gradle
    asnare committed Dec 22, 2015
    Configuration menu
    Copy the full SHA
    f071796 View commit details
    Browse the repository at this point in the history
  2. Remove Optional<...> from *Configuration constructors where actual pr…

    …operty is not optional. This is to make the constructor args match the reality of the config params, such that we can later use JavaDoc for the documentation. Default values are set using the defaultValue annotation, which is however not interpreted by the deserializer. We will later fix this using a custom deserializer in Jackson.
    friso committed Dec 22, 2015
    Configuration menu
    Copy the full SHA
    aaedd85 View commit details
    Browse the repository at this point in the history
  3. Merge branch 'dev/many-to-many' of github.com:divolte/divolte-collect…

    …or into dev/many-to-many
    friso committed Dec 22, 2015
    Configuration menu
    Copy the full SHA
    8ab4af7 View commit details
    Browse the repository at this point in the history
  4. Merge branch 'master' into dev/many-to-many

    # Conflicts:
    #	src/main/java/io/divolte/server/kafka/KafkaFlusher.java
    #	src/main/java/io/divolte/server/kafka/KafkaFlushingPool.java
    #	src/main/resources/reference.conf
    asnare committed Dec 22, 2015
    Configuration menu
    Copy the full SHA
    c0d8153 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    15d057b View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    dfc32c8 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    96b30bc View commit details
    Browse the repository at this point in the history

Commits on Dec 23, 2015

  1. Configuration menu
    Copy the full SHA
    b3d0423 View commit details
    Browse the repository at this point in the history

Commits on Dec 24, 2015

  1. Configuration menu
    Copy the full SHA
    166f6b3 View commit details
    Browse the repository at this point in the history
  2. Start the test server automatically after construction.

    All users of this class did this anyway.
    asnare committed Dec 24, 2015
    Configuration menu
    Copy the full SHA
    18e83a6 View commit details
    Browse the repository at this point in the history
  3. Avoid empty blocks.

    asnare committed Dec 24, 2015
    Configuration menu
    Copy the full SHA
    262562f View commit details
    Browse the repository at this point in the history
  4. Refactor test configurations.

     - Eliminate a bunch of empty configuration files.
     - Change test layout to let tests run against the default configuration with the test server, or a base convenience configuration.
    asnare committed Dec 24, 2015
    Configuration menu
    Copy the full SHA
    d52b832 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    df31645 View commit details
    Browse the repository at this point in the history
  6. Ensure the prefix for a browser source has a trailing '/'.

    The browser source requires this or the tracking script won't correctly locate the event end-point.
    asnare committed Dec 24, 2015
    Configuration menu
    Copy the full SHA
    8c88f76 View commit details
    Browse the repository at this point in the history
  7. Tweak log messages.

    No trailing period (.) required.
    asnare committed Dec 24, 2015
    Configuration menu
    Copy the full SHA
    f3beb9b View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    bcfff98 View commit details
    Browse the repository at this point in the history
  9. Convert method to be static.

    It's invoked by the constructor, and constructors shouldn't normally invoke instance methods because they might not expect the instance to be partially constructed.
    asnare committed Dec 24, 2015
    Configuration menu
    Copy the full SHA
    bcbce80 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    e005a13 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    238dc12 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    6a5cda9 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    28f5b0c View commit details
    Browse the repository at this point in the history
  14. Whitespace.

    asnare committed Dec 24, 2015
    Configuration menu
    Copy the full SHA
    98c55b2 View commit details
    Browse the repository at this point in the history

Commits on Dec 29, 2015

  1. Configuration menu
    Copy the full SHA
    bb4cecd View commit details
    Browse the repository at this point in the history
  2. Support for multiple HDFS and Kafka sinks.

    Changes include:
     - Sink configurations now need to supply a factory for creating the sink.
     - The optional GeoIP service is now passed around as such. It's a bit weird why it was nullable in some places before.
     - There's now a central point for loading schemas. This is needed for schema inference with sinks and ensures schemas are only loaded once.
     - Some utilities for collecting streams into Guava's immutable collections.
     - Processing pools are now shutdown by the central server, instead of recursively via the mapper.
    asnare committed Dec 29, 2015
    Configuration menu
    Copy the full SHA
    46951c7 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    f2523a5 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    a50bbab View commit details
    Browse the repository at this point in the history
  5. Mark utility class as final.

    asnare committed Dec 29, 2015
    Configuration menu
    Copy the full SHA
    756b9bc View commit details
    Browse the repository at this point in the history
  6. Remove an unused logger.

    asnare committed Dec 29, 2015
    Configuration menu
    Copy the full SHA
    4f71734 View commit details
    Browse the repository at this point in the history
  7. Fix incompatible-schema detection for shared sinks.

    The associated tests have also been updated; the fixture for the previous test was not correct.
    asnare committed Dec 29, 2015
    Configuration menu
    Copy the full SHA
    6211569 View commit details
    Browse the repository at this point in the history
  8. Remove some deprecated code.

    asnare committed Dec 29, 2015
    Configuration menu
    Copy the full SHA
    18f55ae View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    ec10591 View commit details
    Browse the repository at this point in the history
  10. Remove superseded todo.

    asnare committed Dec 29, 2015
    Configuration menu
    Copy the full SHA
    a784a15 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    1907a66 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    6ed6dbf View commit details
    Browse the repository at this point in the history

Commits on Dec 30, 2015

  1. Configuration menu
    Copy the full SHA
    5551341 View commit details
    Browse the repository at this point in the history
  2. Remove some redundant test dependencies.

    The main code already depends on these artefacts.
    asnare committed Dec 30, 2015
    Configuration menu
    Copy the full SHA
    bf18049 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    adaee4c View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    7346c90 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    6e83889 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    1d21d72 View commit details
    Browse the repository at this point in the history
  7. Add missing copyright header.

    asnare committed Dec 30, 2015
    Configuration menu
    Copy the full SHA
    5632244 View commit details
    Browse the repository at this point in the history
  8. Factor out most of the browser-source initialisation from the server …

    …class.
    
    This allows source initialisation to take place in parallel, a win because it tends to be fairly slow.
    asnare committed Dec 30, 2015
    Configuration menu
    Copy the full SHA
    a4030e8 View commit details
    Browse the repository at this point in the history

Commits on Dec 31, 2015

  1. Update the example configuration to use the new format.

    Somewhat embarrassingly, the mapping hadn't been converted to groovy yet. (The mapping hasn't been tested.)
    asnare committed Dec 31, 2015
    Configuration menu
    Copy the full SHA
    d07b719 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    aeb4f86 View commit details
    Browse the repository at this point in the history
  3. Refactor the way source configurations are retrieved from the divolte…

    … configuration for consistency.
    
    Retrieving source configurations is now the same as for sinks. In addition, we now universally throw IllegalArgumentException if the the wrong name and/or type is used. (This is a programming mistake, not a user error, as the javadoc now indicates.)
    asnare committed Dec 31, 2015
    Configuration menu
    Copy the full SHA
    66ca030 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    bf0677c View commit details
    Browse the repository at this point in the history

Commits on Feb 1, 2016

  1. Configuration menu
    Copy the full SHA
    ed789fa View commit details
    Browse the repository at this point in the history
  2. Resolve merge.

    friso committed Feb 1, 2016
    Configuration menu
    Copy the full SHA
    c7389fe View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2016

  1. Reflow some of the streaming.

    Also substitute some method handles.
    asnare committed Mar 4, 2016
    Configuration menu
    Copy the full SHA
    7eeb399 View commit details
    Browse the repository at this point in the history
  2. Merge branch 'master' into dev/many-to-many

    # Conflicts:
    #	build.gradle
    asnare committed Mar 4, 2016
    Configuration menu
    Copy the full SHA
    4a23ec1 View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2016

  1. Configuration menu
    Copy the full SHA
    e5d7ad7 View commit details
    Browse the repository at this point in the history

Commits on May 26, 2016

  1. Configuration menu
    Copy the full SHA
    b8f2183 View commit details
    Browse the repository at this point in the history

Commits on Jun 1, 2016

  1. Configuration menu
    Copy the full SHA
    4acc4f0 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e92fa4c View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2016

  1. Configuration menu
    Copy the full SHA
    e07c0e0 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    4231060 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    7cd5d0f View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    1ef3dbb View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    7c88f89 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    d4bc1a3 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    e079f7e View commit details
    Browse the repository at this point in the history
  8. Remove redundant type bounds.

    asnare committed Jun 16, 2016
    Configuration menu
    Copy the full SHA
    f313109 View commit details
    Browse the repository at this point in the history
  9. Avoid Optional.get(); the intent here is that the reference is availa…

    …ble, and if it's not that's exceptional.
    asnare committed Jun 16, 2016
    Configuration menu
    Copy the full SHA
    2a8476d View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    af3d333 View commit details
    Browse the repository at this point in the history

Commits on Jun 24, 2016

  1. Configuration menu
    Copy the full SHA
    9adf153 View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2016

  1. Configuration menu
    Copy the full SHA
    00a342b View commit details
    Browse the repository at this point in the history

Commits on Jul 27, 2016

  1. Configuration menu
    Copy the full SHA
    7336bc4 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2016

  1. Configuration menu
    Copy the full SHA
    253cb7b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    1f0e2f4 View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2016

  1. Configuration menu
    Copy the full SHA
    4574240 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    8b700bf View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    64febcc View commit details
    Browse the repository at this point in the history

Commits on Aug 4, 2016

  1. Configuration menu
    Copy the full SHA
    0f2df5d View commit details
    Browse the repository at this point in the history
  2. Update link in comment to refer to the documentation for the specific…

    … version of Kafka that we're using.
    asnare committed Aug 4, 2016
    Configuration menu
    Copy the full SHA
    288e3ba View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    44d3a07 View commit details
    Browse the repository at this point in the history
  4. Bump documentation copyright.

    asnare committed Aug 4, 2016
    Configuration menu
    Copy the full SHA
    8eb2b62 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    29ecdb9 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    49146e0 View commit details
    Browse the repository at this point in the history
  7. Capitalize list items.

    asnare committed Aug 4, 2016
    Configuration menu
    Copy the full SHA
    7e2fc06 View commit details
    Browse the repository at this point in the history
  8. Render HTML as code block.

    asnare committed Aug 4, 2016
    Configuration menu
    Copy the full SHA
    6342043 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    469f1fb View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    d9497bc View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2016

  1. Update the mapping documentation.

    Changes include Sphinx markup, and updates to reflect that we now support more than a single mapping.
    asnare committed Aug 5, 2016
    Configuration menu
    Copy the full SHA
    ecc4af2 View commit details
    Browse the repository at this point in the history
  2. Sphinx markup for the example strings for user-agent components.

    Plus a few typo corrections.
    asnare committed Aug 5, 2016
    Configuration menu
    Copy the full SHA
    57f695b View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    970f36e View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    0339e9f View commit details
    Browse the repository at this point in the history
  5. GeoNames hyperlink reference.

    asnare committed Aug 5, 2016
    Configuration menu
    Copy the full SHA
    6e0cbe0 View commit details
    Browse the repository at this point in the history
  6. Minor wording changes.

    asnare committed Aug 5, 2016
    Configuration menu
    Copy the full SHA
    13bf817 View commit details
    Browse the repository at this point in the history
  7. Whitespace.

    asnare committed Aug 5, 2016
    Configuration menu
    Copy the full SHA
    4d3475f View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    db44b39 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    e73fe6b View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    b385b04 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    fe5d59b View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2016

  1. Configuration menu
    Copy the full SHA
    9005a31 View commit details
    Browse the repository at this point in the history