[pull] master from mozilla:master

.circleci/config.yml

@@ -28,7 +28,12 @@ jobs:
     - checkout
     - run:
         name: Install dependencies
-        command: sudo pip install mkdocs markdown-include
+        command: |
+          sudo pip install \
+            mkdocs \
+            mkdocs-material \
+            markdown-include \
+            mkdocs-awesome-pages-plugin
     - add_ssh_keys:
         fingerprints:
         - "84:b0:66:dd:ec:68:b1:45:9d:5d:66:fd:4a:4f:1b:57"

GRAVEYARD.md

@@ -1,5 +1,11 @@
 # Link to code removed from this repository
 
+- 2021-03-12 (available until [commit `c05fbab`](https://github.com/mozilla/gcp-ingestion/commit/c05fbabcb44c5a8a290be311a87951728cf587b6))
+  - Remove support for Account Ecosystem Telemetry (AET);
+    see [Bug 1697602](https://bugzilla.mozilla.org/show_bug.cgi?id=1697602)
+  - Also removes `ParseLogEntry` which transforms a Google Cloud Logging
+    (`Stackdriver`) `LogEntry` message into one compatible with structured
+    ingestion, which could be relevant for future use cases.
 - 2020-06-01 (available until [commit `f1d6464`](https://github.com/mozilla/gcp-ingestion/commit/f1d646442b8c1fcd63202ebca91363979b5b2ae2))
   - Remove `DeduplicateByDocumentId` transform, which was intended for use with
     the backfill from `heka` data, but did not perform well and was never used

README.md

@@ -21,6 +21,7 @@ There are currently four components:
 For more information, see [the documentation](https://mozilla.github.io/gcp-ingestion).
 
 Java 11 support is a work in progress for the Beam Java SDK, so this project requires
-Java 8 and will likely fail to compile using newer versions of the JDK.
+Java 8. Maven has been configured to compile for Java 8 when using newer versions of the
+JDK, but support is only guaranteed for JDK 8.
 To manage multiple local JDKs, consider [jenv](https://www.jenv.be/) and the
 `jenv enable-plugin maven` command.

docs/architecture/decoder_service_specification.md

@@ -6,6 +6,8 @@ in the Structured Ingestion pipeline.
 ## Data Flow
 
 1. Consume messages from Google Cloud PubSub raw topic
+1. Deduplicate message by `uri` (which generally contains `docId`)
+   - Disabled for stub-installer, which does not include a UUID in the URI
 1. Perform GeoIP lookup and drop `x_forwarded_for` and `remote_addr` and
    optionally `geo_city` based on population
 1. Parse the `uri` attribute to determine document type, etc.
@@ -16,16 +18,12 @@ in the Structured Ingestion pipeline.
 1. Validate the schema of the body
 1. Extract user agent information and drop `user_agent`
 1. Add metadata fields to message
-1. Deduplicate message by `docId`
-   - Generate `docId` for submission types that don't have one
 1. Write message to PubSub decoded topic based on `namespace` and `docType`
 
 ### Implementation
 
 The above steps will be executed as a single Apache Beam job that can accept
 either a streaming input from PubSub or a batch input from Cloud Storage.
-Message deduplication will be done by checking for the presence of ids as keys
-in Cloud Memory Store (managed Redis).
 
 ### Decoding Errors
 
@@ -118,13 +116,12 @@ method, to ensure ack'd messages are fully delivered.
 
 ### Deduplication
 
-Each `docId` will be allowed through "at least once", and only be
+Each `uri` will be allowed through "at least once", and only be
 rejected as a duplicate if we have completed delivery of a message with the
-same `docId`. Duplicates will be considered errors and sent to the error topic.
+same `uri`. We assume that each `uri` contains a UUID that uniquely identifies
+the document.
 "Exactly once" semantics can be applied to derived data sets using SQL in
 BigQuery, and GroupByKey in Beam and Spark.
 
-Note that deduplication is only provided with a "best effort" quality of service.
-In the ideal case, we hold 24 hours of history for seen document IDs, but that
-buffer is allowed to degrade to a shorter time window when the pipeline is under
-high load.
+Note that deduplication is only provided with a "best effort" quality of service
+using a 10 minute window.

docs/architecture/diagram.mmd

@@ -9,21 +9,16 @@ graph TD
 
 f1(Producers) --> k1(Ingestion Edge)
 k1 --> p1(Raw Topic)
-k1 --> p1aet(AET Raw Topic)
 k1 --> p99(...)
 
 subgraph Pipeline Family: Structured
     p1 --> k2(Raw Sink)
     k2 --> c1(BigQuery)
     p1 --> d2(Decoder)
-    p1aet --> d2aet(AET Decoder)
-    d2aet -. Decrypt keys on startup .- m2(Cloud KMS)
     d2 --> p2(Decoded Topic)
     d2 -.-> p2err(Error Topic)
     p2err --> d3err(Error Sink)
     d3err --> berr(BigQuery)
-    d2aet --> p2
-    d2aet -.-> p2err
     p2 --> d3(Live Sink)
     d3 --> b1(BigQuery)
     p2 --> k3(Decoded Sink)
@@ -37,9 +32,6 @@ subgraph .
    p99 --> d99(...)
 end
 
-d5 -. Mark document as seen .-> m1
-m1(Cloud Memorystore) -. Document already seen? .-> d2
-
 subgraph Colors
     d(Dataflow jobs are green)
     k(Kubernetes services are magenta)

docs/architecture/overview.md

@@ -15,12 +15,8 @@ This document specifies the architecture for GCP Ingestion as a whole.
   `BigQuery`
 - The Dataflow `Decoder` job decodes messages from the PubSub `Raw Topic` to
   the PubSub `Decoded Topic`
-  - Also checks for existence of `document_id`s in
-    `Cloud Memorystore` in order to deduplicate messages
-- The Dataflow `AET Decoder` job provides all the functionality of the `Decoder`
-  with additional decryption handling for Account Ecosystem Telemetry pings
-- The Dataflow `Republisher` job reads messages from the PubSub `Decoded Topic`,
-  marks them as seen in `Cloud Memorystore` and republishes them to various
+- The Dataflow `Republisher` job reads messages from the PubSub `Decoded Topic`
+  and republishes them to various
   lower volume derived topics including `Monitoring Sample Topics` and
   `Per DocType Topics`
 - The Kubernetes `Decoded Sink` job copies messages from the PubSub `Decoded Topic`
@@ -70,37 +66,14 @@ This document specifies the architecture for GCP Ingestion as a whole.
   1. Produce `normalized_` variants of select attributes
   1. Inject `normalized_` attributes at the top level and other select
      attributes into a nested `metadata` top level key in `payload`
-- Should deduplicate messages based on the `document_id` attribute using
-  `Cloud MemoryStore`
+- Should deduplicate messages based on the `uri` attribute
   - Must ensure at least once delivery, so deduplication is only "best effort"
-  - Should delay deduplication to the latest possible stage of the pipeline
-    to minimize the time window between an ID being marked as seen in
-    `Republisher` and it being checked in `Decoder`
 - Must send messages rejected by transforms to a configurable error destination
   - Must allow error destination in BigQuery
 
-### AET Decoder
-
-The AET (Account Ecosystem Telemetry) Decoder is a modified version of the
-Decoder with the following properties:
-
-- The raw topic that feeds the AET Decoder must not be sent anywhere else;
-  the AET Decoder needs to either successfully decrypt or sanitize all AET
-  identifiers
-- Must load private keys from an encrypted blob in GCS
-- Must call Cloud KMS at startup to decrypt keys and store these only in memory
-- Must remove or redact all AET `ecosystem_anon_id` values from the payload before
-  passing to any durable output, including errors
-- Must have access restricted to a limited set of operators to avoid exposing private keys
-- Encrypted fields must be JOSE JWE objects in Compact Serialization form
-
 ### Republisher
 
 - Must copy messages from PubSub topics to PubSub topics
-- Must attempt to publish the `document_id` of each consumed message to
-  `Cloud MemoryStore`
-  - ID publishing should be "best effort" but must not prevent the message
-    proceeding to further steps in case of errors reaching `Cloud MemoryStore`
 - Must ack messages read from PubSub after they are delivered to all
   matching destinations
 - Must not ack messages read from PubSub before they are delivered to all

docs/architecture/pain_points.md

@@ -45,8 +45,8 @@ is to specify mapping at template creation time.
 Does not use standard client library.
 
 Does not expose an output of delivered messages, which is needed for at least
-once delivery with deduplication. Current workaround is to get delivered
-messages from a subscription to the output PubSub topic.
+once delivery with deduplication. Current workaround is to use the deduplication
+available via `PubsubIO.read()`.
 
 Uses HTTPS JSON API, which increases message payload size vs protobuf by 25%
 for base64 encoding and causes some messages to exceed the 10MB request size
@@ -57,9 +57,9 @@ limit that otherwise would not.
 ## Templates
 
 Does not support repeated parameters via `ValueProvider<List<...>>`, as
-described in [Dataflow Java SDK #632]
+described in [Dataflow Java SDK #632].
 
-[googlecloudplatform/dataflowjavasdk#632]: https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues/632
+[dataflow java sdk #632]: https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues/632
 
 # PubSub