-
Notifications
You must be signed in to change notification settings - Fork 78
Allow for multiple endpoints and many-to-many mappings of endpoints to sinks #115
Commits on Nov 5, 2015
-
Start of work to support many-to-many sources/sinks/mappings.
Changes include: - A new configuration layout for supporting multiple sources, sinks and mappings. - No more enqueuing delay. Queues tend to either be full or empty, and are really only effective at smoothing jitter. Spinning to delay doesn't really help, so we just drop the events immediately instead of blocking. - No more session-binning strategy for HDFS. In practice we didn't use this, and it's complicated to do properly when multiple sources (with their own idea of sessions) might be feeding into a sink. For now, the code only supports the existing single browser-based source and HDFS/Kafka sinks. Configuration validation is also not yet implemented.
Configuration menu - View commit details
-
Copy full SHA for 453e6d5 - Browse repository at this point
Copy the full SHA 453e6d5View commit details
Commits on Nov 6, 2015
-
Configuration menu - View commit details
-
Copy full SHA for f8ad26c - Browse repository at this point
Copy the full SHA f8ad26cView commit details
Commits on Nov 23, 2015
-
Configuration menu - View commit details
-
Copy full SHA for 0acf80e - Browse repository at this point
Copy the full SHA 0acf80eView commit details
Commits on Nov 24, 2015
-
Configuration menu - View commit details
-
Copy full SHA for d32f4ea - Browse repository at this point
Copy the full SHA d32f4eaView commit details -
Do not throw RuntimeException on MappingException, but instead catch …
…mapping exceptions and turn them into useful error messages and leave configuratio invalid.
Configuration menu - View commit details
-
Copy full SHA for e51eabb - Browse repository at this point
Copy the full SHA e51eabbView commit details -
Configuration menu - View commit details
-
Copy full SHA for c00ed4e - Browse repository at this point
Copy the full SHA c00ed4eView commit details
Commits on Dec 4, 2015
-
Configuration menu - View commit details
-
Copy full SHA for 04d9b2b - Browse repository at this point
Copy the full SHA 04d9b2bView commit details
Commits on Dec 7, 2015
-
Configuration menu - View commit details
-
Copy full SHA for 7fa5f73 - Browse repository at this point
Copy the full SHA 7fa5f73View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4ffeff7 - Browse repository at this point
Copy the full SHA 4ffeff7View commit details
Commits on Dec 22, 2015
-
Merge branch 'master' into dev/many-to-many
# Conflicts: # build.gradle
Configuration menu - View commit details
-
Copy full SHA for f071796 - Browse repository at this point
Copy the full SHA f071796View commit details -
Remove Optional<...> from *Configuration constructors where actual pr…
…operty is not optional. This is to make the constructor args match the reality of the config params, such that we can later use JavaDoc for the documentation. Default values are set using the defaultValue annotation, which is however not interpreted by the deserializer. We will later fix this using a custom deserializer in Jackson.
Configuration menu - View commit details
-
Copy full SHA for aaedd85 - Browse repository at this point
Copy the full SHA aaedd85View commit details -
Merge branch 'dev/many-to-many' of github.com:divolte/divolte-collect…
…or into dev/many-to-many
Configuration menu - View commit details
-
Copy full SHA for 8ab4af7 - Browse repository at this point
Copy the full SHA 8ab4af7View commit details -
Merge branch 'master' into dev/many-to-many
# Conflicts: # src/main/java/io/divolte/server/kafka/KafkaFlusher.java # src/main/java/io/divolte/server/kafka/KafkaFlushingPool.java # src/main/resources/reference.conf
Configuration menu - View commit details
-
Copy full SHA for c0d8153 - Browse repository at this point
Copy the full SHA c0d8153View commit details -
Configuration menu - View commit details
-
Copy full SHA for 15d057b - Browse repository at this point
Copy the full SHA 15d057bView commit details -
Configuration menu - View commit details
-
Copy full SHA for dfc32c8 - Browse repository at this point
Copy the full SHA dfc32c8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 96b30bc - Browse repository at this point
Copy the full SHA 96b30bcView commit details
Commits on Dec 23, 2015
-
Configuration menu - View commit details
-
Copy full SHA for b3d0423 - Browse repository at this point
Copy the full SHA b3d0423View commit details
Commits on Dec 24, 2015
-
Configuration menu - View commit details
-
Copy full SHA for 166f6b3 - Browse repository at this point
Copy the full SHA 166f6b3View commit details -
Start the test server automatically after construction.
All users of this class did this anyway.
Configuration menu - View commit details
-
Copy full SHA for 18e83a6 - Browse repository at this point
Copy the full SHA 18e83a6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 262562f - Browse repository at this point
Copy the full SHA 262562fView commit details -
- Eliminate a bunch of empty configuration files. - Change test layout to let tests run against the default configuration with the test server, or a base convenience configuration.
Configuration menu - View commit details
-
Copy full SHA for d52b832 - Browse repository at this point
Copy the full SHA d52b832View commit details -
Configuration menu - View commit details
-
Copy full SHA for df31645 - Browse repository at this point
Copy the full SHA df31645View commit details -
Ensure the prefix for a browser source has a trailing '/'.
The browser source requires this or the tracking script won't correctly locate the event end-point.
Configuration menu - View commit details
-
Copy full SHA for 8c88f76 - Browse repository at this point
Copy the full SHA 8c88f76View commit details -
Configuration menu - View commit details
-
Copy full SHA for f3beb9b - Browse repository at this point
Copy the full SHA f3beb9bView commit details -
Configuration menu - View commit details
-
Copy full SHA for bcfff98 - Browse repository at this point
Copy the full SHA bcfff98View commit details -
It's invoked by the constructor, and constructors shouldn't normally invoke instance methods because they might not expect the instance to be partially constructed.
Configuration menu - View commit details
-
Copy full SHA for bcbce80 - Browse repository at this point
Copy the full SHA bcbce80View commit details -
Configuration menu - View commit details
-
Copy full SHA for e005a13 - Browse repository at this point
Copy the full SHA e005a13View commit details -
Configuration menu - View commit details
-
Copy full SHA for 238dc12 - Browse repository at this point
Copy the full SHA 238dc12View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6a5cda9 - Browse repository at this point
Copy the full SHA 6a5cda9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 28f5b0c - Browse repository at this point
Copy the full SHA 28f5b0cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 98c55b2 - Browse repository at this point
Copy the full SHA 98c55b2View commit details
Commits on Dec 29, 2015
-
Configuration menu - View commit details
-
Copy full SHA for bb4cecd - Browse repository at this point
Copy the full SHA bb4cecdView commit details -
Support for multiple HDFS and Kafka sinks.
Changes include: - Sink configurations now need to supply a factory for creating the sink. - The optional GeoIP service is now passed around as such. It's a bit weird why it was nullable in some places before. - There's now a central point for loading schemas. This is needed for schema inference with sinks and ensures schemas are only loaded once. - Some utilities for collecting streams into Guava's immutable collections. - Processing pools are now shutdown by the central server, instead of recursively via the mapper.
Configuration menu - View commit details
-
Copy full SHA for 46951c7 - Browse repository at this point
Copy the full SHA 46951c7View commit details -
Configuration menu - View commit details
-
Copy full SHA for f2523a5 - Browse repository at this point
Copy the full SHA f2523a5View commit details -
Configuration menu - View commit details
-
Copy full SHA for a50bbab - Browse repository at this point
Copy the full SHA a50bbabView commit details -
Configuration menu - View commit details
-
Copy full SHA for 756b9bc - Browse repository at this point
Copy the full SHA 756b9bcView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4f71734 - Browse repository at this point
Copy the full SHA 4f71734View commit details -
Fix incompatible-schema detection for shared sinks.
The associated tests have also been updated; the fixture for the previous test was not correct.
Configuration menu - View commit details
-
Copy full SHA for 6211569 - Browse repository at this point
Copy the full SHA 6211569View commit details -
Configuration menu - View commit details
-
Copy full SHA for 18f55ae - Browse repository at this point
Copy the full SHA 18f55aeView commit details -
Configuration menu - View commit details
-
Copy full SHA for ec10591 - Browse repository at this point
Copy the full SHA ec10591View commit details -
Configuration menu - View commit details
-
Copy full SHA for a784a15 - Browse repository at this point
Copy the full SHA a784a15View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1907a66 - Browse repository at this point
Copy the full SHA 1907a66View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6ed6dbf - Browse repository at this point
Copy the full SHA 6ed6dbfView commit details
Commits on Dec 30, 2015
-
Configuration menu - View commit details
-
Copy full SHA for 5551341 - Browse repository at this point
Copy the full SHA 5551341View commit details -
Remove some redundant test dependencies.
The main code already depends on these artefacts.
Configuration menu - View commit details
-
Copy full SHA for bf18049 - Browse repository at this point
Copy the full SHA bf18049View commit details -
Configuration menu - View commit details
-
Copy full SHA for adaee4c - Browse repository at this point
Copy the full SHA adaee4cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7346c90 - Browse repository at this point
Copy the full SHA 7346c90View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6e83889 - Browse repository at this point
Copy the full SHA 6e83889View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1d21d72 - Browse repository at this point
Copy the full SHA 1d21d72View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5632244 - Browse repository at this point
Copy the full SHA 5632244View commit details -
Factor out most of the browser-source initialisation from the server …
…class. This allows source initialisation to take place in parallel, a win because it tends to be fairly slow.
Configuration menu - View commit details
-
Copy full SHA for a4030e8 - Browse repository at this point
Copy the full SHA a4030e8View commit details
Commits on Dec 31, 2015
-
Update the example configuration to use the new format.
Somewhat embarrassingly, the mapping hadn't been converted to groovy yet. (The mapping hasn't been tested.)
Configuration menu - View commit details
-
Copy full SHA for d07b719 - Browse repository at this point
Copy the full SHA d07b719View commit details -
Configuration menu - View commit details
-
Copy full SHA for aeb4f86 - Browse repository at this point
Copy the full SHA aeb4f86View commit details -
Refactor the way source configurations are retrieved from the divolte…
… configuration for consistency. Retrieving source configurations is now the same as for sinks. In addition, we now universally throw IllegalArgumentException if the the wrong name and/or type is used. (This is a programming mistake, not a user error, as the javadoc now indicates.)
Configuration menu - View commit details
-
Copy full SHA for 66ca030 - Browse repository at this point
Copy the full SHA 66ca030View commit details -
Configuration menu - View commit details
-
Copy full SHA for bf0677c - Browse repository at this point
Copy the full SHA bf0677cView commit details
Commits on Feb 1, 2016
-
Configuration menu - View commit details
-
Copy full SHA for ed789fa - Browse repository at this point
Copy the full SHA ed789faView commit details -
Configuration menu - View commit details
-
Copy full SHA for c7389fe - Browse repository at this point
Copy the full SHA c7389feView commit details
Commits on Mar 4, 2016
-
Configuration menu - View commit details
-
Copy full SHA for 7eeb399 - Browse repository at this point
Copy the full SHA 7eeb399View commit details -
Merge branch 'master' into dev/many-to-many
# Conflicts: # build.gradle
Configuration menu - View commit details
-
Copy full SHA for 4a23ec1 - Browse repository at this point
Copy the full SHA 4a23ec1View commit details
Commits on Mar 8, 2016
-
Configuration menu - View commit details
-
Copy full SHA for e5d7ad7 - Browse repository at this point
Copy the full SHA e5d7ad7View commit details
Commits on May 26, 2016
-
Configuration menu - View commit details
-
Copy full SHA for b8f2183 - Browse repository at this point
Copy the full SHA b8f2183View commit details
Commits on Jun 1, 2016
-
Configuration menu - View commit details
-
Copy full SHA for 4acc4f0 - Browse repository at this point
Copy the full SHA 4acc4f0View commit details -
Configuration menu - View commit details
-
Copy full SHA for e92fa4c - Browse repository at this point
Copy the full SHA e92fa4cView commit details
Commits on Jun 16, 2016
-
Configuration menu - View commit details
-
Copy full SHA for e07c0e0 - Browse repository at this point
Copy the full SHA e07c0e0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4231060 - Browse repository at this point
Copy the full SHA 4231060View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7cd5d0f - Browse repository at this point
Copy the full SHA 7cd5d0fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1ef3dbb - Browse repository at this point
Copy the full SHA 1ef3dbbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7c88f89 - Browse repository at this point
Copy the full SHA 7c88f89View commit details -
Configuration menu - View commit details
-
Copy full SHA for d4bc1a3 - Browse repository at this point
Copy the full SHA d4bc1a3View commit details -
Configuration menu - View commit details
-
Copy full SHA for e079f7e - Browse repository at this point
Copy the full SHA e079f7eView commit details -
Configuration menu - View commit details
-
Copy full SHA for f313109 - Browse repository at this point
Copy the full SHA f313109View commit details -
Avoid Optional.get(); the intent here is that the reference is availa…
…ble, and if it's not that's exceptional.
Configuration menu - View commit details
-
Copy full SHA for 2a8476d - Browse repository at this point
Copy the full SHA 2a8476dView commit details -
Configuration menu - View commit details
-
Copy full SHA for af3d333 - Browse repository at this point
Copy the full SHA af3d333View commit details
Commits on Jun 24, 2016
-
Configuration menu - View commit details
-
Copy full SHA for 9adf153 - Browse repository at this point
Copy the full SHA 9adf153View commit details
Commits on Jul 22, 2016
-
Configuration menu - View commit details
-
Copy full SHA for 00a342b - Browse repository at this point
Copy the full SHA 00a342bView commit details
Commits on Jul 27, 2016
-
Configuration menu - View commit details
-
Copy full SHA for 7336bc4 - Browse repository at this point
Copy the full SHA 7336bc4View commit details
Commits on Aug 2, 2016
-
Configuration menu - View commit details
-
Copy full SHA for 253cb7b - Browse repository at this point
Copy the full SHA 253cb7bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1f0e2f4 - Browse repository at this point
Copy the full SHA 1f0e2f4View commit details
Commits on Aug 3, 2016
-
Configuration menu - View commit details
-
Copy full SHA for 4574240 - Browse repository at this point
Copy the full SHA 4574240View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8b700bf - Browse repository at this point
Copy the full SHA 8b700bfView commit details -
Configuration menu - View commit details
-
Copy full SHA for 64febcc - Browse repository at this point
Copy the full SHA 64febccView commit details
Commits on Aug 4, 2016
-
Configuration menu - View commit details
-
Copy full SHA for 0f2df5d - Browse repository at this point
Copy the full SHA 0f2df5dView commit details -
Update link in comment to refer to the documentation for the specific…
… version of Kafka that we're using.
Configuration menu - View commit details
-
Copy full SHA for 288e3ba - Browse repository at this point
Copy the full SHA 288e3baView commit details -
Introduce event-flows, and mention that we now support many-to-many m…
…appings in Divolte.
Configuration menu - View commit details
-
Copy full SHA for 44d3a07 - Browse repository at this point
Copy the full SHA 44d3a07View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8eb2b62 - Browse repository at this point
Copy the full SHA 8eb2b62View commit details -
Configuration menu - View commit details
-
Copy full SHA for 29ecdb9 - Browse repository at this point
Copy the full SHA 29ecdb9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 49146e0 - Browse repository at this point
Copy the full SHA 49146e0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7e2fc06 - Browse repository at this point
Copy the full SHA 7e2fc06View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6342043 - Browse repository at this point
Copy the full SHA 6342043View commit details -
Configuration menu - View commit details
-
Copy full SHA for 469f1fb - Browse repository at this point
Copy the full SHA 469f1fbView commit details -
Clarify that a specified schema is ignored by a mapping unless the sc…
…ript is specified.
Configuration menu - View commit details
-
Copy full SHA for d9497bc - Browse repository at this point
Copy the full SHA d9497bcView commit details
Commits on Aug 5, 2016
-
Update the mapping documentation.
Changes include Sphinx markup, and updates to reflect that we now support more than a single mapping.
Configuration menu - View commit details
-
Copy full SHA for ecc4af2 - Browse repository at this point
Copy the full SHA ecc4af2View commit details -
Sphinx markup for the example strings for user-agent components.
Plus a few typo corrections.
Configuration menu - View commit details
-
Copy full SHA for 57f695b - Browse repository at this point
Copy the full SHA 57f695bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 970f36e - Browse repository at this point
Copy the full SHA 970f36eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0339e9f - Browse repository at this point
Copy the full SHA 0339e9fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6e0cbe0 - Browse repository at this point
Copy the full SHA 6e0cbe0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 13bf817 - Browse repository at this point
Copy the full SHA 13bf817View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4d3475f - Browse repository at this point
Copy the full SHA 4d3475fView commit details -
Configuration menu - View commit details
-
Copy full SHA for db44b39 - Browse repository at this point
Copy the full SHA db44b39View commit details -
Configuration menu - View commit details
-
Copy full SHA for e73fe6b - Browse repository at this point
Copy the full SHA e73fe6bView commit details -
Configuration menu - View commit details
-
Copy full SHA for b385b04 - Browse repository at this point
Copy the full SHA b385b04View commit details -
Configuration menu - View commit details
-
Copy full SHA for fe5d59b - Browse repository at this point
Copy the full SHA fe5d59bView commit details
Commits on Aug 9, 2016
-
Configuration menu - View commit details
-
Copy full SHA for 9005a31 - Browse repository at this point
Copy the full SHA 9005a31View commit details