Integrate elasticsearch

Author: callmekatootie

README.md

@@ -5,41 +5,76 @@
 - node 12.x
 - npm 6.x
 - docker
+- elasticsearch 6.x
+
+## Configuration
+
+Configuration for the application is at config/default.js and config/production.js. The following parameters can be set in config files or in env variables:
+
+- LOG_LEVEL: the log level
+- PORT: the server port
+- AUTH_SECRET: TC Authentication secret
+- VALID_ISSUERS: valid issuers for TC authentication
+- PAGE_SIZE: the default pagination limit
+- API_VERSION: the API version
+- AWS_KEY: The AWS access key
+- AWS_SECRET: The AWS secret key
+- AWS_REGION: The Amazon region to use when connecting.
+- DATABASE: The QLDB ledger name
+- AUTH0_URL: Auth0 URL, used to get TC M2M token
+- AUTH0_AUDIENCE: Auth0 audience, used to get TC M2M token
+- TOKEN_CACHE_TIME: Auth0 token cache time, used to get TC M2M token
+- AUTH0_CLIENT_ID: Auth0 client id, used to get TC M2M token
+- AUTH0_CLIENT_SECRET: Auth0 client secret, used to get TC M2M token
+- AUTH0_PROXY_SERVER_URL: Proxy Auth0 URL, used to get TC M2M token
+- GROUP_API_URL: Topcoder Group API URL
+- BUSAPI_URL: Topcoder Bus API URL
+- KAFKA_ERROR_TOPIC: The error topic at which bus api will publish any errors
+- KAFKA_MESSAGE_ORIGINATOR: The originator value for the kafka messages
+- UBAHN_CREATE_TOPIC: Kafka topic for create message
+- UBAHN_UPDATE_TOPIC: Kafka topic for update message
+- UBAHN_DELETE_TOPIC: Kafka topic for delete message
+- ES.HOST: Elasticsearch host
+- ES.AWS_REGION: The Amazon region to use when using AWS Elasticsearch service
+- ES.API_VERSION: Elasticsearch API version
+- ES.DOCUMENTS: Elasticsearch index, type and id mapping for resources.
+
+For `ES.DOCUMENTS` configuration, you will find multiple other configurations below it. Each has default values that you can override using the environment variables
 
 ## Local deployment
 
+Setup your Elasticsearch instance and ensure that it is up and running.
+
 1. Visit [this link](https://console.aws.amazon.com/qldb/home?region=us-east-1#gettingStarted), login and create one **ledger** databases named `ubahn-db`
 2. Visit [this link](https://console.aws.amazon.com/iam/home?region=us-east-1#/security_credentials) to download your "Access keys"
 3. Follow *Configuration* section to update config values, like database, aws key/secret etc ..
 4. Goto *UBahn-api*, run `npm i` and `npm run lint`
 5. Import mock data, `node scripts/db/genData.js`, this will create tables and gen some data for test (if you need this)
 6. Startup server `node app.js` or `npm run start`
 
-## Docker
+## Local Deployment with Docker
 
 Make sure all config values are right(aws key and secret), and you can run on local successful, then run below commands
 
-- Run `docker build -t tc/ubahn_api .` to build image
-- Then run `docker run tc/ubahn_api -d` to startup image
+1. Navigate to the directory `docker`
+
+2. Rename the file `sample.api.env` to `api.env`
+
+3. Set the required AUTH0 configurations, AWS credentials and ElasticSearch host in the file `api.env`
+
+4. Once that is done, run the following command
+
+    ```bash
+    docker-compose up
+    ```
+
+5. When you are running the application for the first time, It will take some time initially to download the image and install the dependencies
 
 ## API endpoints verification
 
 1. open postman
 2. import *docs/UBahn_API.postman_collection.json* , *UBahn_ENV.postman_environment.json* and then check endpoints
 
-## Configuration
-
-| key           | system Environment name | description                |
-| ------------- | ----------------------- | -------------------------- |
-| PORT          | PORT                    | the server port            |
-| AUTH_SECRET   | AUTH_SECRET             | the jwt client secret      |
-| VALID_ISSUERS | VALID_ISSUERS           | jwt token issuers          |
-| API_VERSION   |                         | the api prefix version     |
-| AWS_KEY       | AWS_KEY                 | the aws Access key         |
-| AWS_SECRET    | AWS_SECRET              | the aws Access secret      |
-| AWS_REGION    | AWS_REGION              | the aws service region     |
-| DATABASE      | DATABASE                | the aws QLDB database name |
-
 ## Test token
 
 you can use below token to test role and permissions

VERIFICATION.md

@@ -0,0 +1,102 @@
+## Verification
+The verification needs data in ES, there are two ways to populate data to ES.
+
+Please install and run [kibana](https://www.elastic.co/downloads/past-releases/kibana-6-8-0), validate data in ES during the testing, I found this is very useful for developing and testing.
+The data in ES is confusing, please validate data in ES before making a decision success or failure.
+
+#### Using u-bahn-api service
+Start the `u-bahn-api` service and create data with postman.
+As you create data, verify data is read from ES using resource GET methods from postman.
+When create a resource, the postman sets the Id of corresponding resource, so be carefully after create resource
+
+According to the [forum](https://apps.topcoder.com/forums/?module=Thread&threadID=956692&start=0), 
+the u-bahn topics isn't set up in TC bus api, so the posting message to bus API doesn't work currently.
+Fix TC bus api to accept u-bahn topics, please use this approach, this is the easiest way.
+
+
+#### Using u-bahn-es-processor
+
+Publish messages to [u-bahn-es-processor](https://github.com/topcoder-platform/u-bahn-es-processor) through kafka. 
+This is very hard and error prune, but if this is the only way, use this approach.
+
+Follow the instructions in the README file in the `u-bahm-es-processor`, start kafka server, start elasticsearch, initialize Elasticsearch, start `u-bahn-es-processor` app
+
+1. start kafka-console-producer to write messages to `u-bahn.action.create`
+topic:
+  `docker exec -it ubahn-data-processor-es_kafka /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic u-bahn.action.create`
+3. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"user","id":"391a3656-9a01-47d4-8c6d-64b68c44f212","handle":"user"}}`
+4. Watch the app console, It will show message successfully handled.
+5. Get the user from the postman by GET /users or /users/391a3656-9a01-47d4-8c6d-64b68c44f212.
+
+6. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"achievement","userId":"391a3656-9a01-47d4-8c6d-64b68c44f212","achievementsProviderId":"c77326d8-ef16-4be0-b844-d5c384b7bb8b","name":"achievement","uri":"https://google.com","certifierId":"b8726ca1-557e-4502-8f9b-25044b9c123d","certifiedDate":"2019-07-08T00:00:00.000Z"}}`
+7. Watch the app console, It will show message successfully handled.
+8. Get the achievement from the postman.
+
+9. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"achievementprovider","id":"c77326d8-ef16-4be0-b844-d5c384b7bb8b","name":"achievementprovider"}}`
+10. Watch the app console, It will show message successfully handled.
+11. Get the achievementprovider from the postman.
+
+12. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"attributegroup","id":"720c34f9-0fd4-46fd-9293-4a8cfdcd3e96","organizationId":"017733ad-4704-4c7e-ae60-36b3332731df","name":"attributegroup"}}`
+13. Watch the app console, It will show message successfully handled.
+14. Get the attributegroup from the postman.
+
+15. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"externalprofile","userId":"391a3656-9a01-47d4-8c6d-64b68c44f212","organizationId":"017733ad-4704-4c7e-ae60-36b3332731df","uri":"https:google.com"}}`
+16. Watch the app console, It will show message successfully handled.
+17. Get the externalprofile from the postman.
+
+18. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"organization","id":"017733ad-4704-4c7e-ae60-36b3332731df","name":"organization"}}`
+19. Watch the app console, It will show message successfully handled.
+20. Get the organization from the postman.
+
+21. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"role","id":"188446f1-02dc-4fc7-b74e-ab7ea3033a57","name":"role"}}`
+22. Watch the app console, It will show message successfully handled.
+23. Get the role from the postman.
+
+24. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"skill","id":"8a8c8d3a-9165-4dae-8a8c-f828cbe01d5d","skillProviderId":"63061b84-9784-4b71-b695-4a777eeb7601","externalId":"ba395d36-6ce8-4bd1-9d6c-754f0389abcb","uri":"https://google.com","name":"skill"}}`
+25. Watch the app console, It will show message successfully handled.
+26. Get the skill from the postman.
+
+27. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"skillprovider","id":"63061b84-9784-4b71-b695-4a777eeb7601","name":"skillprovider"}}`
+28. Watch the app console, It will show message successfully handled.
+29. Get the skillprovider from the postman.
+
+30. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"userattribute","userId":"391a3656-9a01-47d4-8c6d-64b68c44f212","attributeId":"b5a50f73-08e2-43d1-a78a-4652f15d950e","value":"userattribute"}}`
+31. Watch the app console, It will show message successfully handled.
+32. Get the userattribute from the postman.
+
+33. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"userrole","userId":"391a3656-9a01-47d4-8c6d-64b68c44f212","roleId":"188446f1-02dc-4fc7-b74e-ab7ea3033a57"}}`
+34. Watch the app console, It will show message successfully handled.
+35. Get the userrole from the postman.
+
+36. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"userskill","userId":"391a3656-9a01-47d4-8c6d-64b68c44f212","skillId":"8a8c8d3a-9165-4dae-8a8c-f828cbe01d5d","metricValue":"userskill","certifierId":"7cf786d9-a8c0-48ed-a7cc-09dcf91d904c","certifiedDate":"2019-07-08T00:00:00.000Z"}}`
+37. Watch the app console, It will show message successfully handled.
+38. Get the userskill from the postman.
+
+39. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"attribute","id":"b5a50f73-08e2-43d1-a78a-4652f15d950e","name":"attribute", "attributeGroupId": "720c34f9-0fd4-46fd-9293-4a8cfdcd3e96"}}`
+40. Watch the app console, It will show message successfully handled.
+41. Get the attribute from the postman.
+
+42. These are full set of data for one user.
+Get users with enrich=true, verify all data above are returned
+
+43. Verify query parameters in each resource are works based on the swagger doc.
+When setting query parameter to the non-existing value, verify the app tries to get the data from QLDB in the console.
+
+44. By changing id values from above data, create another users or other resources.
+
+45. Test paging, query filtering with multiple resources.
+
+46. Please verify all specifications in the project.

config/default.js

@@ -5,13 +5,92 @@
 module.exports = {
   LOG_LEVEL: process.env.LOG_LEVEL || 'debug',
   PORT: process.env.PORT || 3001,
+
   AUTH_SECRET: process.env.AUTH_SECRET || 'CLIENT_SECRET',
   VALID_ISSUERS: process.env.VALID_ISSUERS ? process.env.VALID_ISSUERS.replace(/\\"/g, '')
     : '["https://topcoder-dev.auth0.com/", "https://api.topcoder.com"]',
-  API_VERSION: 'api/1.0',
+
+  PAGE_SIZE: process.env.PAGE_SIZE || 20,
+  API_VERSION: process.env.API_VERSION || 'api/1.0',
 
   AWS_KEY: process.env.AWS_KEY,
   AWS_SECRET: process.env.AWS_SECRET,
   AWS_REGION: process.env.AWS_REGION || 'us-east-1',
-  DATABASE: process.env.DATABASE || 'ubahn-db'
+  DATABASE: process.env.DATABASE || 'ubahn-db',
+
+  AUTH0_URL: process.env.AUTH0_URL,
+  AUTH0_AUDIENCE: process.env.AUTH0_AUDIENCE,
+  TOKEN_CACHE_TIME: process.env.TOKEN_CACHE_TIME,
+  AUTH0_CLIENT_ID: process.env.AUTH0_CLIENT_ID,
+  AUTH0_CLIENT_SECRET: process.env.AUTH0_CLIENT_SECRET,
+  AUTH0_PROXY_SERVER_URL: process.env.AUTH0_PROXY_SERVER_URL,
+
+  GROUP_API_URL: process.env.GROUP_API_URL || 'https://api.topcoder-dev.com/v5/groups',
+  BUSAPI_URL: process.env.BUSAPI_URL || 'https://api.topcoder-dev.com/v5',
+
+  KAFKA_ERROR_TOPIC: process.env.KAFKA_ERROR_TOPIC || 'common.error.reporting',
+  KAFKA_MESSAGE_ORIGINATOR: process.env.KAFKA_MESSAGE_ORIGINATOR || 'u-bahn-api',
+
+  // topics
+  UBAHN_CREATE_TOPIC: process.env.UBAHN_CREATE_TOPIC || 'u-bahn.action.create',
+  UBAHN_UPDATE_TOPIC: process.env.UBAHN_UPDATE_TOPIC || 'u-bahn.action.update',
+  UBAHN_DELETE_TOPIC: process.env.UBAHN_DELETE_TOPIC || 'u-bahn.action.delete',
+
+  // ElasticSearch
+  ES: {
+    HOST: process.env.ES_HOST || 'localhost:9200',
+    AWS_REGION: process.env.AWS_REGION || 'us-east-1', // AWS Region to be used if we use AWS ES
+    API_VERSION: process.env.ES_API_VERSION || '6.8',
+    // es mapping: _index, _type, _id
+    DOCUMENTS: {
+      achievementprovider: {
+        index: process.env.ACHIEVEMENT_PROVIDER_INDEX || 'achievement_provider',
+        type: '_doc'
+      },
+      attribute: {
+        index: process.env.ATTRIBUTE_INDEX || 'attribute',
+        type: '_doc'
+      },
+      attributegroup: {
+        index: process.env.ATTRIBUTE_GROUP_INDEX || 'attribute_group',
+        type: '_doc'
+      },
+      organization: {
+        index: process.env.ORGANIZATION_INDEX || 'organization',
+        type: '_doc'
+      },
+      role: {
+        index: process.env.ROLE_INDEX || 'role',
+        type: '_doc'
+      },
+      skill: {
+        index: process.env.SKILL_INDEX || 'skill',
+        type: '_doc'
+      },
+      skillprovider: {
+        index: process.env.SKILL_PROVIDER_INDEX || 'skill_provider',
+        type: '_doc'
+      },
+      user: {
+        index: process.env.USER_INDEX || 'user',
+        type: '_doc'
+      },
+      // sub resources under user
+      achievement: {
+        userField: process.env.USER_ACHIEVEMENT_PROPERTY_NAME || 'achievements'
+      },
+      externalprofile: {
+        userField: process.env.USER_EXTERNALPROFILE_PROPERTY_NAME || 'externalProfiles'
+      },
+      userattribute: {
+        userField: process.env.USER_ATTRIBUTE_PROPERTY_NAME || 'attributes'
+      },
+      userrole: {
+        userField: process.env.USER_ROLE_PROPERTY_NAME || 'roles'
+      },
+      userskill: {
+        userField: process.env.USER_SKILL_PROPERTY_NAME || 'skills'
+      }
+    }
+  }
 }

docker/Dockerfile

@@ -0,0 +1,12 @@
+# Use the base image with Node.js 12
+FROM node:12
+
+# Copy the current directory into the Docker image
+COPY . /ubahn_api
+
+# Set working directory for future use
+WORKDIR /ubahn_api
+
+# Install the dependencies from package.json
+RUN npm install
+CMD npm start

docker/docker-compose.yml

@@ -0,0 +1,11 @@
+version: '3'
+services:
+  ubahn_api:
+    image: ubahn_api:latest
+    build:
+      context: ../
+      dockerfile: docker/Dockerfile
+    env_file:
+      - api.env
+    ports:
+      - "3001:3001"

docker/sample.api.env

@@ -0,0 +1,9 @@
+AWS_ACCESS_KEY_ID=<AWS Access Key ID>
+AWS_SECRET_ACCESS_KEY=<AWS Secret Access Key>
+ES_HOST=<ES Host Endpoint>
+
+AUTH0_URL=<AUTH0 URL>
+AUTH0_AUDIENCE=<AUTH0 Audience>
+TOKEN_CACHE_TIME=500000
+AUTH0_CLIENT_ID=<AUTH0 Client ID>
+AUTH0_CLIENT_SECRET=<AUTH0 Client Secret>