Merge pull request #39 from Financial-Times/feature/UPPSF-1425-new-props

Feature/uppsf 1425 new props
Financial-Times · Jun 29, 2020 · f3a33b4 · f3a33b4
2 parents 7dd6501 + 5827f47
commit f3a33b4
Show file tree

Hide file tree

Showing 16 changed files with 211 additions and 71 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -1,7 +1,7 @@
 version: 2.1
 
 orbs:
-  snyk: snyk/snyk@0.0.8
+  snyk: snyk/snyk@0.0.10
 
 jobs:
   build:

diff --git a/README.md b/README.md
@@ -2,23 +2,28 @@
 
 [![Circle CI](https://circleci.com/gh/Financial-Times/content-rw-elasticsearch/tree/master.png?style=shield)](https://circleci.com/gh/Financial-Times/content-rw-elasticsearch/tree/master)[![Go Report Card](https://goreportcard.com/badge/github.com/Financial-Times/content-rw-elasticsearch)](https://goreportcard.com/report/github.com/Financial-Times/content-rw-elasticsearch) [![Coverage Status](https://coveralls.io/repos/github/Financial-Times/content-rw-elasticsearch/badge.svg)](https://coveralls.io/github/Financial-Times/content-rw-elasticsearch)
 
-
 ## Introduction
+
 Indexes V2 content in Elasticsearch for use by SAPI V1
 
 ## Project Local Execution
 
 Before executing any of the proposed ways run
-```
+
+```sh
 make install
 ```
+
 to install external tools needed and then
-```
+
+```sh
 make all
 ```
+
 to run tests and a clean build of the project.
 
 ---
+
 **NOTE**
 
 Each time you modify any file under the `configs` directory, please run `make generate` in order to regenerate
@@ -27,44 +32,64 @@ the `statik` package which embeds the files in the binary.
 ---
 
 ### Docker Compose
+
 `docker-compose` is used to provide application external components:
+
 * Elasticsearch
 
 and to start the application itself.
 
 **Step 1.** Build application docker image
-```
+
+```sh
+export GITHUB_USERNAME="<username>"
+export GITHUB_TOKEN="<token>"
+
 docker-compose build --no-cache app
 ```
-**Step 2.** Run Elasticsearch 
-```
+
+**Step 2.** Run Elasticsearch
+
+```sh
 docker-compose up -d es
 ```
+
 **Step 3.** Create Elasticsearch index mapping
-```
+
+```sh
 cd <project_home>
 curl -X PUT http://localhost:9200/ft/ -d @configs/referenceSchema.json
 ```
+
 **Step 4.** Run application
-```
+
+```sh
 docker-compose up -d app
 ```
+
 **Step 5.** Check health endpoint in browser at `http://localhost:8080/__health`
 
 ### Application CLI
+
 **Step 1.** Build project and run tests
-```
+
+```sh
 make all
 ```
+
 or just build project
-```
+
+```sh
 make build-readonly
 ```
+
 **Step 2.** Run the binary (using the `--help` flag to see the available optional arguments):
-```
+
+```sh
 <project_home>/content-rw-elasticsearch [--help]
 ```
-```
+
+```txt
 Options:
       --app-system-code                System Code of the application (env $APP_SYSTEM_CODE) (default "content-rw-elasticsearch")
       --app-name                       Application name (env $APP_NAME) (default "content-rw-elasticsearch")
@@ -81,39 +106,40 @@ Options:
       --kafka-concurrent-processing    Whether the consumer uses concurrent processing for the messages (env $KAFKA_CONCURRENT_PROCESSING)
       --public-concordances-endpoint   Endpoint to concord ids with (env $PUBLIC_CONCORDANCES_ENDPOINT) (default "http://public-concordances-api:8080")
       --base-api-url                   Base API URL (env $BASE_API_URL) (default "https://api.ft.com/")
-```                    
-Whether the consumer uses concurrent processing for the messages ($KAFKA_CONCURRENT_PROCESSING)
+```
 
+Whether the consumer uses concurrent processing for the messages ($KAFKA_CONCURRENT_PROCESSING)
 
 ## Build and deployment
 
 * Built by Docker Hub on merge to master: [coco/content-rw-elasticsearch](https://hub.docker.com/r/coco/content-rw-elasticsearch/)
 * CI provided by CircleCI: [content-rw-elasticsearch](https://circleci.com/gh/Financial-Times/content-rw-elasticsearch)
 
 ## Healthchecks
+
 Admin endpoints are:
 
-    `/__gtg`
+`/__gtg`
 
 Returns 503 if any if the checks executed at the /__health endpoint returns false
 
-    `/__health`
-    
+`/__health`
+
 There are several checks performed:
+
 * Elasticsearch cluster connectivity
 * Elasticsearch cluster health
 * Elastic schema validation
 * Kafka queue topic check
 
+`/__health-details`
 
-    `/__health-details`
-
 Shows ES cluster health details
 
-    `/__build-info` 
-
+`/__build-info`
 
 ## Other information
+
 An example of event structure is here [testdata/exampleEnrichedContentModel.json](messaging/testdata/exampleEnrichedContentModel.json)
 
 The reference mappings for Elasticsearch are found here [configs/referenceSchema.json](configs/referenceSchema.json)
diff --git a/configs/referenceSchema.json b/configs/referenceSchema.json
@@ -38,6 +38,15 @@
           "type": "string",
           "include_in_all": true
         },
+        "scoop": {
+          "type": "boolean"
+        },
+        "canBeDistributed": {
+          "type": "string"
+        },
+        "canBeSyndicated": {
+          "type": "string"
+        },
         "byline": {
           "type": "string",
           "fields": {
@@ -798,6 +807,15 @@
           "type": "string",
           "include_in_all": true
         },
+        "scoop": {
+          "type": "boolean"
+        },
+        "canBeDistributed": {
+          "type": "string"
+        },
+        "canBeSyndicated": {
+          "type": "string"
+        },
         "byline": {
           "type": "string",
           "fields": {
@@ -1573,6 +1591,15 @@
           "type": "string",
           "include_in_all": true
         },
+        "scoop": {
+          "type": "boolean"
+        },
+        "canBeDistributed": {
+          "type": "string"
+        },
+        "canBeSyndicated": {
+          "type": "string"
+        },
         "byline": {
           "type": "string",
           "fields": {
@@ -2330,6 +2357,15 @@
           "type": "string",
           "include_in_all": true
         },
+        "scoop": {
+          "type": "boolean"
+        },
+        "canBeDistributed": {
+          "type": "string"
+        },
+        "canBeSyndicated": {
+          "type": "string"
+        },
         "byline": {
           "type": "string",
           "fields": {
@@ -3087,6 +3123,15 @@
           "type": "string",
           "include_in_all": true
         },
+        "scoop": {
+          "type": "boolean"
+        },
+        "canBeDistributed": {
+          "type": "string"
+        },
+        "canBeSyndicated": {
+          "type": "string"
+        },
         "byline": {
           "type": "string",
           "fields": {
@@ -3844,6 +3889,15 @@
           "type": "string",
           "include_in_all": true
         },
+        "scoop": {
+          "type": "boolean"
+        },
+        "canBeDistributed": {
+          "type": "string"
+        },
+        "canBeSyndicated": {
+          "type": "string"
+        },
         "byline": {
           "type": "string",
           "fields": {
@@ -4587,4 +4641,4 @@
       }
     }
   }
-}
+}
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -1,7 +1,11 @@
 version: '3'
 services:
   app:
-    build: .
+    build:
+      context: .
+      args:
+        GITHUB_USERNAME: ${GITHUB_USERNAME}
+        GITHUB_TOKEN: ${GITHUB_TOKEN}
     image: content-rw-elasticsearch:local
     container_name: content-rw-elasticsearch
     environment:
@@ -29,4 +33,3 @@ volumes:
 
 networks:
   common:
-
diff --git a/helm/content-rw-elasticsearch/app-configs/content-rw-elasticsearch-reindexer_delivery.yaml b/helm/content-rw-elasticsearch/app-configs/content-rw-elasticsearch-reindexer_delivery.yaml
@@ -7,3 +7,4 @@ env:
   KAFKA_CONCURRENT_PROCESSING: true
   PUBLIC_CONCORDANCES_ENDPOINT: http://public-concordances-api:8080
   INTERNAL_CONTENT_API_URL: http://internal-content-api:8080
+  APP_SYSTEM_CODE: content-rw-elasticsearch-reindexer
diff --git a/helm/content-rw-elasticsearch/app-configs/content-rw-elasticsearch_delivery.yaml b/helm/content-rw-elasticsearch/app-configs/content-rw-elasticsearch_delivery.yaml
@@ -7,3 +7,4 @@ env:
   KAFKA_CONCURRENT_PROCESSING: false
   PUBLIC_CONCORDANCES_ENDPOINT: http://public-concordances-api:8080
   INTERNAL_CONTENT_API_URL: http://internal-content-api:8080
+  APP_SYSTEM_CODE: content-rw-elasticsearch
diff --git a/helm/content-rw-elasticsearch/templates/deployment.yaml b/helm/content-rw-elasticsearch/templates/deployment.yaml
@@ -1,22 +1,22 @@
 apiVersion: extensions/v1beta1
 kind: Deployment
 metadata:
-  name: {{ .Values.service.name }} 
+  name: {{ .Values.service.name }}
   labels:
     chart: "{{ .Chart.Name | trunc 63 }}"
-    chartVersion: "{{ .Chart.Version | trunc 63 }}" 
-    visualize: "true" 
-    app: {{ .Values.service.name }} 
+    chartVersion: "{{ .Chart.Version | trunc 63 }}"
+    visualize: "true"
+    app: {{ .Values.service.name }}
 spec:
-  replicas: {{ .Values.replicaCount }} 
-  selector: 
+  replicas: {{ .Values.replicaCount }}
+  selector:
     matchLabels:
       app: {{ .Values.service.name }}
   template:
     metadata:
       labels:
         app: {{ .Values.service.name }}
-        visualize: "true" 
+        visualize: "true"
     spec:
       affinity:
         podAntiAffinity:
@@ -29,25 +29,27 @@ spec:
                 - {{ .Values.service.name }}
             topologyKey: "kubernetes.io/hostname"
       containers:
-      - name: {{ .Values.service.name }} 
+      - name: {{ .Values.service.name }}
         image: "{{ .Values.image.repository }}:{{ .Chart.Version }}"
         imagePullPolicy: {{ .Values.image.pullPolicy }}
-        env: 
-        - name: ELASTICSEARCH_SAPI_ENDPOINT 
-          valueFrom: 
-            configMapKeyRef: 
-              name: global-config 
+        env:
+        - name: ELASTICSEARCH_SAPI_ENDPOINT
+          valueFrom:
+            configMapKeyRef:
+              name: global-config
               key: aws.content.elasticsearch.endpoint
-        - name: KAFKA_CONSUMER_GROUP 
-          value: "k8s-{{ .Values.service.name }}" 
-        - name: KAFKA_TOPIC 
+        - name: ELASTICSEARCH_SAPI_INDEX
+          value: "{{ .Values.env.ELASTICSEARCH_SAPI_INDEX }}"
+        - name: KAFKA_CONSUMER_GROUP
+          value: "k8s-{{ .Values.service.name }}"
+        - name: KAFKA_TOPIC
           value: {{ .Values.env.KAFKA_TOPIC }}
-        - name: KAFKA_CONCURRENT_PROCESSING 
+        - name: KAFKA_CONCURRENT_PROCESSING
           value: "{{ .Values.env.KAFKA_CONCURRENT_PROCESSING }}"
-        - name: KAFKA_PROXY_ADDR 
-          valueFrom: 
-            configMapKeyRef: 
-              name: global-config 
+        - name: KAFKA_PROXY_ADDR
+          valueFrom:
+            configMapKeyRef:
+              name: global-config
               key: kafka.proxy.url.with.protocol
         - name: AWS_ACCESS_KEY_ID
           valueFrom:
@@ -68,18 +70,17 @@ spec:
             configMapKeyRef:
               name: global-config
               key: api.host.with.protocol
-        ports: 
-        - containerPort: 8080 
-        livenessProbe: 
-          tcpSocket: 
-            port: 8080 
-          initialDelaySeconds: 10 
-        readinessProbe: 
-          httpGet: 
-            path: "/__gtg" 
-            port: 8080 
-          initialDelaySeconds: 15 
-          periodSeconds: 30 
-        resources: 
+        ports:
+        - containerPort: 8080
+        livenessProbe:
+          tcpSocket:
+            port: 8080
+          initialDelaySeconds: 10
+        readinessProbe:
+          httpGet:
+            path: "/__gtg"
+            port: 8080
+          initialDelaySeconds: 15
+          periodSeconds: 30
+        resources:
 {{ toYaml .Values.resources | indent 12 }}
-