Skip to content
This repository has been archived by the owner on Apr 22, 2022. It is now read-only.

Allow for multiple endpoints and many-to-many mappings of endpoints to sinks #115

Merged
merged 103 commits into from
Aug 9, 2016
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
103 commits
Select commit Hold shift + click to select a range
453e6d5
Start of work to support many-to-many sources/sinks/mappings.
asnare Nov 5, 2015
f8ad26c
Fix the default value for automatic page-view events.
asnare Nov 6, 2015
0acf80e
Favor ImmutableProperties over getter with defensive copy.
friso Nov 23, 2015
d32f4ea
Use deprecation annotation instead of comment.
friso Nov 24, 2015
e51eabb
Do not throw RuntimeException on MappingException, but instead catch …
friso Nov 24, 2015
c00ed4e
Extract some methods for readability.
friso Nov 24, 2015
04d9b2b
Validators for DivolteConfiguration.
friso Dec 4, 2015
7fa5f73
Reordering of methods.
friso Dec 7, 2015
4ffeff7
Flatten ClientSideCookieHandler and super class into one class.
friso Dec 7, 2015
f071796
Merge branch 'master' into dev/many-to-many
asnare Dec 22, 2015
aaedd85
Remove Optional<...> from *Configuration constructors where actual pr…
friso Dec 22, 2015
8ab4af7
Merge branch 'dev/many-to-many' of github.com:divolte/divolte-collect…
friso Dec 22, 2015
c0d8153
Merge branch 'master' into dev/many-to-many
asnare Dec 22, 2015
15d057b
Store source ID and affinity in items on the processing queues.
friso Dec 22, 2015
dfc32c8
Resolve conflict on Kafka flushing code.
friso Dec 22, 2015
96b30bc
Merge branch 'master' into dev/many-to-many
asnare Dec 22, 2015
b3d0423
Use .isEmpty() instead of comparing size to 0.
asnare Dec 23, 2015
166f6b3
During tests, fail early if the test server configuration isn't valid.
asnare Dec 24, 2015
18e83a6
Start the test server automatically after construction.
asnare Dec 24, 2015
262562f
Avoid empty blocks.
asnare Dec 24, 2015
d52b832
Refactor test configurations.
asnare Dec 24, 2015
df31645
Add a test for the /ping endpoint.
asnare Dec 24, 2015
8c88f76
Ensure the prefix for a browser source has a trailing '/'.
asnare Dec 24, 2015
f3beb9b
Tweak log messages.
asnare Dec 24, 2015
bcfff98
Start supporting multiple browser endpoints.
asnare Dec 24, 2015
bcbce80
Convert method to be static.
asnare Dec 24, 2015
e005a13
Merge branch 'master' into dev/many-to-many
asnare Dec 24, 2015
238dc12
Minor optimisations during flushing: pre-size the arrays we're stream…
asnare Dec 24, 2015
6a5cda9
Nullability annotations and enforcement.
asnare Dec 24, 2015
28f5b0c
Static references are normally UPPER_CASE.
asnare Dec 24, 2015
98c55b2
Whitespace.
asnare Dec 24, 2015
bb4cecd
No need for either of these tests to reference the Kafka sink.
asnare Dec 29, 2015
46951c7
Support for multiple HDFS and Kafka sinks.
asnare Dec 29, 2015
f2523a5
Fix test configurations, now that we elide disabled sinks.
asnare Dec 29, 2015
a50bbab
Some basic tests for multiple sinks.
asnare Dec 29, 2015
756b9bc
Mark utility class as final.
asnare Dec 29, 2015
4f71734
Remove an unused logger.
asnare Dec 29, 2015
6211569
Fix incompatible-schema detection for shared sinks.
asnare Dec 29, 2015
18f55ae
Remove some deprecated code.
asnare Dec 29, 2015
ec10591
Additional logging during startup.
asnare Dec 29, 2015
a784a15
Remove superseded todo.
asnare Dec 29, 2015
1907a66
Specialise event forwarding for common cases.
asnare Dec 29, 2015
6ed6dbf
Instantiate all mappings during startup, not just one of them.
asnare Dec 29, 2015
5551341
Support many-to-many mappings between sources and mappings.
asnare Dec 30, 2015
bf18049
Remove some redundant test dependencies.
asnare Dec 30, 2015
adaee4c
Factor out versions where we have multiple artefacts that use the sam…
asnare Dec 30, 2015
7346c90
Merge the test suites for testing various source and sink scenarios.
asnare Dec 30, 2015
6e83889
Fix a bug where the schema registry threw an exception while handling…
asnare Dec 30, 2015
1d21d72
Add some tests for various source/mapping/sink configurations.
asnare Dec 30, 2015
5632244
Add missing copyright header.
asnare Dec 30, 2015
a4030e8
Factor out most of the browser-source initialisation from the server …
asnare Dec 30, 2015
d07b719
Update the example configuration to use the new format.
asnare Dec 31, 2015
aeb4f86
Merge branch 'master' into dev/many-to-many
asnare Dec 31, 2015
66ca030
Refactor the way source configurations are retrieved from the divolte…
asnare Dec 31, 2015
bf0677c
Annotate some methods whose overrides should invoke super.
asnare Dec 31, 2015
ed789fa
Move all mapping onto a single thread pool.
friso Feb 1, 2016
c7389fe
Resolve merge.
friso Feb 1, 2016
7eeb399
Reflow some of the streaming.
asnare Mar 4, 2016
4a23ec1
Merge branch 'master' into dev/many-to-many
asnare Mar 4, 2016
e5d7ad7
Merge branch 'master' into dev/many-to-many
asnare Mar 8, 2016
b8f2183
Merge branch 'master' into dev/many-to-many
asnare May 26, 2016
4acc4f0
Merge branch 'dev/version-bumps' into dev/many-to-many
asnare Jun 1, 2016
e92fa4c
Merge branch 'master' into dev/many-to-many
asnare Jun 1, 2016
e07c0e0
Consistent null annotation and default handling during configuration …
asnare Jun 16, 2016
4231060
Fix a typo in the .toString() implementation.
asnare Jun 16, 2016
7cd5d0f
Speed up tests by disabling HDFS support unless it's needed for the t…
asnare Jun 16, 2016
1ef3dbb
Merge branch 'master' into dev/many-to-many
asnare Jun 16, 2016
7c88f89
Refactor to use a builder for the list of errors.
asnare Jun 16, 2016
d4bc1a3
Return the value instead of assigning to intermediate reference.
asnare Jun 16, 2016
e079f7e
Remove unnecessary exceptions from throw specification.
asnare Jun 16, 2016
f313109
Remove redundant type bounds.
asnare Jun 16, 2016
2a8476d
Avoid Optional.get(); the intent here is that the reference is availa…
asnare Jun 16, 2016
af3d333
Merge branch 'master' into dev/many-to-many
asnare Jun 16, 2016
9adf153
Merge branch 'master' into dev/many-to-many
asnare Jun 24, 2016
00a342b
Merge branch 'master' into dev/many-to-many
asnare Jul 22, 2016
7336bc4
Merge branch 'master' into dev/many-to-many
asnare Jul 27, 2016
253cb7b
Merge branch 'master' into dev/many-to-many
asnare Aug 2, 2016
1f0e2f4
Merge branch 'master' into dev/many-to-many
asnare Aug 2, 2016
4574240
Merge branch 'master' into dev/many-to-many
asnare Aug 3, 2016
8b700bf
Update the configuration file we bundle with the RPM to use the new l…
asnare Aug 3, 2016
64febcc
Update the 'Getting Started' documentation to match the new configura…
asnare Aug 3, 2016
0f2df5d
Ask sphinx to be picky about problems.
asnare Aug 4, 2016
288e3ba
Update link in comment to refer to the documentation for the specific…
asnare Aug 4, 2016
44d3a07
Introduce event-flows, and mention that we now support many-to-many m…
asnare Aug 4, 2016
8eb2b62
Bump documentation copyright.
asnare Aug 4, 2016
29ecdb9
Update the configuration reference to describe the new way Divolte is…
asnare Aug 4, 2016
49146e0
Update links to refer to the Avro specification version that we use.
asnare Aug 4, 2016
7e2fc06
Capitalize list items.
asnare Aug 4, 2016
6342043
Render HTML as code block.
asnare Aug 4, 2016
469f1fb
Use Sphinx semantic markup where possible.
asnare Aug 4, 2016
d9497bc
Clarify that a specified schema is ignored by a mapping unless the sc…
asnare Aug 4, 2016
ecc4af2
Update the mapping documentation.
asnare Aug 5, 2016
57f695b
Sphinx markup for the example strings for user-agent components.
asnare Aug 5, 2016
970f36e
Metro codes aren't an ISO thing.
asnare Aug 5, 2016
0339e9f
Latitude and longitude aren't really English.
asnare Aug 5, 2016
6e0cbe0
GeoNames hyperlink reference.
asnare Aug 5, 2016
13bf817
Minor wording changes.
asnare Aug 5, 2016
4d3475f
Whitespace.
asnare Aug 5, 2016
db44b39
Refresh the deployment documentation.
asnare Aug 5, 2016
e73fe6b
Merge branch 'master' into dev/many-to-many
asnare Aug 5, 2016
b385b04
Merge branch 'master' into dev/many-to-many
asnare Aug 5, 2016
fe5d59b
Fix some failing Selenium tests.
asnare Aug 5, 2016
9005a31
Call out the implicit default sources/mapping/sinks with a heading.
asnare Aug 9, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
34 changes: 19 additions & 15 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -75,12 +75,18 @@ repositories {
}

dependencies {
// Define some key versions for components that we use lots of artifacts from.
// Note: we refer to the Avro documentation in our own documentation.
def avroVersion = '1.8.1'
def hadoopVersion = '2.7.2'
def jacksonVersion = '2.8.1'
def slf4jVersion = '1.7.21'

compile group: 'io.divolte', name: 'divolte-schema', version: version
compile group: 'io.undertow', name: 'undertow-core', version: '1.4.0.Final'
compile group: 'com.typesafe', name: 'config', version: '1.3.0'
compile group: 'com.google.guava', name: 'guava', version: '19.0'
// Note: we refer to the Avro documentation in our own documentation.
compile group: 'org.apache.avro', name: 'avro', version: '1.8.1'
compile group: 'org.apache.avro', name: 'avro', version: avroVersion

/*
* We package the Avro Tools to provide an easy way to view Avro files
Expand All @@ -91,14 +97,14 @@ dependencies {
* run and have a working tojson command. The other commands weren't fully
* tested with these deps.
*/
compile group: 'org.apache.avro', name: 'avro-tools', version: '1.8.1', classifier: 'nodeps'
compile group: 'org.apache.avro', name: 'trevni-core', version: '1.8.1'
compile group: 'org.apache.avro', name: 'avro-mapred', version: '1.8.1'
compile group: 'org.apache.avro', name: 'avro-tools', version: avroVersion, classifier: 'nodeps'
compile group: 'org.apache.avro', name: 'trevni-core', version: avroVersion
compile group: 'org.apache.avro', name: 'avro-mapred', version: avroVersion

compile (group: 'org.apache.hadoop', name:'hadoop-common', version: '2.7.2') {
compile (group: 'org.apache.hadoop', name:'hadoop-common', version: hadoopVersion) {
exclude group: 'jline', module: 'jline'
}
compile group: 'org.apache.hadoop', name:'hadoop-hdfs', version: '2.7.2'
compile group: 'org.apache.hadoop', name:'hadoop-hdfs', version: hadoopVersion
compile (group: 'net.sf.uadetector', name: 'uadetector-core', version: '0.9.22') {
exclude group: 'com.google.code.findbugs', module: 'jsr305'
}
Expand All @@ -116,9 +122,10 @@ dependencies {
compile group: 'org.codehaus.groovy', name:'groovy', version: '2.4.7', classifier: 'indy'
compile group: 'net.sf.jopt-simple', name:'jopt-simple', version: '5.0.2'
compile group: 'com.jayway.jsonpath', name: 'json-path', version: '2.2.0'
compile group: 'com.fasterxml.jackson.core', name:'jackson-databind', version: '2.8.1'
compile group: 'com.fasterxml.jackson.datatype', name:'jackson-datatype-jdk8', version: '2.8.1'
compile group: 'com.fasterxml.jackson.module', name:'jackson-module-parameter-names', version: '2.8.1'
compile group: 'com.fasterxml.jackson.core', name:'jackson-databind', version: jacksonVersion
compile group: 'com.fasterxml.jackson.datatype', name:'jackson-datatype-jdk8', version: jacksonVersion
compile group: 'com.fasterxml.jackson.datatype', name:'jackson-datatype-guava', version: jacksonVersion
compile group: 'com.fasterxml.jackson.module', name:'jackson-module-parameter-names', version: jacksonVersion
compile group: 'com.jasonclawson', name: 'jackson-dataformat-hocon', version: '1.1.0'

// Used for configuration validation
Expand All @@ -127,16 +134,13 @@ dependencies {

// We use the SLF4J API. At runtime, this is LogBack.
// (We also force any dependencies that use Log4J to go via SLF4J.)
compile group: 'org.slf4j', name: 'slf4j-api', version: '1.7.21'
compile group: 'org.slf4j', name: 'slf4j-api', version: slf4jVersion
runtime group: 'ch.qos.logback', name: 'logback-classic', version: '1.1.7'
runtime group: 'org.slf4j', name: 'log4j-over-slf4j', version: '1.7.21'
runtime group: 'org.slf4j', name: 'log4j-over-slf4j', version: slf4jVersion

testCompile group: 'junit', name: 'junit', version: '4.12'
testCompile group: 'org.hamcrest', name: 'hamcrest-all', version: '1.3'
testCompile group: 'org.mockito', name: 'mockito-all', version: '1.10.19'
testCompile group: 'com.fasterxml.jackson.core', name:'jackson-databind', version: '2.6.3'
testCompile group: 'com.fasterxml.jackson.module', name:'jackson-module-parameter-names', version: '2.6.3'
testCompile group: 'com.fasterxml.jackson.datatype', name:'jackson-datatype-jdk8', version: '2.6.3'
testCompile group: 'com.saucelabs', name:'sauce_junit', version: '2.1.21'

testCompile group: 'org.seleniumhq.selenium', name:'selenium-java', version: '2.53.1'
Expand Down
6 changes: 4 additions & 2 deletions docs/conf.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# -*- coding: utf-8 -*-
#
# Copyright 2014 GoDataDriven B.V.
# Copyright 2016 GoDataDriven B.V.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -53,7 +53,7 @@

# General information about the project.
project = u'Divolte'
copyright = u'2015, GoDataDriven'
copyright = u'2016, GoDataDriven'

# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
Expand Down Expand Up @@ -103,6 +103,8 @@
# If true, keep warnings as "system message" paragraphs in the built documents.
#keep_warnings = False

# If true, Sphinx will warn about all references where the target cannot be found.
#nitpicky = False

# -- Options for HTML output ----------------------------------------------

Expand Down