Merge branch 'develop' into health-check

Author: callmekatootie

README.md

@@ -5,41 +5,79 @@
 - node 12.x
 - npm 6.x
 - docker
+- elasticsearch 6.x
+
+## Configuration
+
+Configuration for the application is at config/default.js and config/production.js. The following parameters can be set in config files or in env variables:
+
+- LOG_LEVEL: the log level
+- PORT: the server port
+- AUTH_SECRET: TC Authentication secret
+- VALID_ISSUERS: valid issuers for TC authentication
+- PAGE_SIZE: the default pagination limit
+- API_VERSION: the API version
+- AWS_ACCESS_KEY_ID: The AWS access key
+- AWS_SECRET_ACCESS_KEY: The AWS secret key
+- AWS_REGION: The Amazon region to use when connecting.
+- DATABASE: The QLDB ledger name
+- AUTH0_URL: Auth0 URL, used to get TC M2M token
+- AUTH0_AUDIENCE: Auth0 audience, used to get TC M2M token
+- TOKEN_CACHE_TIME: Auth0 token cache time, used to get TC M2M token
+- AUTH0_CLIENT_ID: Auth0 client id, used to get TC M2M token
+- AUTH0_CLIENT_SECRET: Auth0 client secret, used to get TC M2M token
+- AUTH0_PROXY_SERVER_URL: Proxy Auth0 URL, used to get TC M2M token
+- GROUP_API_URL: Topcoder Group API URL
+- BUSAPI_URL: Topcoder Bus API URL
+- KAFKA_ERROR_TOPIC: The error topic at which bus api will publish any errors
+- KAFKA_MESSAGE_ORIGINATOR: The originator value for the kafka messages
+- UBAHN_CREATE_TOPIC: Kafka topic for create message
+- UBAHN_UPDATE_TOPIC: Kafka topic for update message
+- UBAHN_DELETE_TOPIC: Kafka topic for delete message
+- ES.HOST: Elasticsearch host
+- ES.API_VERSION: Elasticsearch API version
+- ES.DOCUMENTS: Elasticsearch index, type and id mapping for resources.
+
+For `ES.DOCUMENTS` configuration, you will find multiple other configurations below it. Each has default values that you can override using the environment variables
 
 ## Local deployment
 
+Setup your Elasticsearch instance and ensure that it is up and running.
+
 1. Visit [this link](https://console.aws.amazon.com/qldb/home?region=us-east-1#gettingStarted), login and create one **ledger** databases named `ubahn-db`
 2. Visit [this link](https://console.aws.amazon.com/iam/home?region=us-east-1#/security_credentials) to download your "Access keys"
 3. Follow *Configuration* section to update config values, like database, aws key/secret etc ..
 4. Goto *UBahn-api*, run `npm i` and `npm run lint`
 5. Import mock data, `node scripts/db/genData.js`, this will create tables and gen some data for test (if you need this)
 6. Startup server `node app.js` or `npm run start`
 
-## Docker
+## Working with mock data
+
+You can use the scripts `npm run insert-data` (and `npm run delete-data`) to insert mock data (and delete mock data respectively). The data is inserted into QLDB and Elasticsearch. You need to setup the configurations beforehand and also start the elasticsearch instance before you run these scripts
+
+## Local Deployment with Docker
 
 Make sure all config values are right(aws key and secret), and you can run on local successful, then run below commands
 
-- Run `docker build -t tc/ubahn_api .` to build image
-- Then run `docker run tc/ubahn_api -d` to startup image
+1. Navigate to the directory `docker`
+
+2. Rename the file `sample.api.env` to `api.env`
+
+3. Set the required AUTH0 configurations, AWS credentials and ElasticSearch host in the file `api.env`
+
+4. Once that is done, run the following command
+
+    ```bash
+    docker-compose up
+    ```
+
+5. When you are running the application for the first time, It will take some time initially to download the image and install the dependencies
 
 ## API endpoints verification
 
 1. open postman
 2. import *docs/UBahn_API.postman_collection.json* , *UBahn_ENV.postman_environment.json* and then check endpoints
 
-## Configuration
-
-| key           | system Environment name | description                |
-| ------------- | ----------------------- | -------------------------- |
-| PORT          | PORT                    | the server port            |
-| AUTH_SECRET   | AUTH_SECRET             | the jwt client secret      |
-| VALID_ISSUERS | VALID_ISSUERS           | jwt token issuers          |
-| API_VERSION   |                         | the api prefix version     |
-| AWS_KEY       | AWS_KEY                 | the aws Access key         |
-| AWS_SECRET    | AWS_SECRET              | the aws Access secret      |
-| AWS_REGION    | AWS_REGION              | the aws service region     |
-| DATABASE      | DATABASE                | the aws QLDB database name |
-
 ## Test token
 
 you can use below token to test role and permissions

VERIFICATION.md

@@ -0,0 +1,102 @@
+## Verification
+The verification needs data in ES, there are two ways to populate data to ES.
+
+Please install and run [kibana](https://www.elastic.co/downloads/past-releases/kibana-6-8-0), validate data in ES during the testing, I found this is very useful for developing and testing.
+The data in ES is confusing, please validate data in ES before making a decision success or failure.
+
+#### Using u-bahn-api service
+Start the `u-bahn-api` service and create data with postman.
+As you create data, verify data is read from ES using resource GET methods from postman.
+When create a resource, the postman sets the Id of corresponding resource, so be carefully after create resource
+
+According to the [forum](https://apps.topcoder.com/forums/?module=Thread&threadID=956692&start=0), 
+the u-bahn topics isn't set up in TC bus api, so the posting message to bus API doesn't work currently.
+Fix TC bus api to accept u-bahn topics, please use this approach, this is the easiest way.
+
+
+#### Using u-bahn-es-processor
+
+Publish messages to [u-bahn-es-processor](https://github.com/topcoder-platform/u-bahn-es-processor) through kafka. 
+This is very hard and error prune, but if this is the only way, use this approach.
+
+Follow the instructions in the README file in the `u-bahm-es-processor`, start kafka server, start elasticsearch, initialize Elasticsearch, start `u-bahn-es-processor` app
+
+1. start kafka-console-producer to write messages to `u-bahn.action.create`
+topic:
+  `docker exec -it ubahn-data-processor-es_kafka /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic u-bahn.action.create`
+3. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"user","id":"391a3656-9a01-47d4-8c6d-64b68c44f212","handle":"user"}}`
+4. Watch the app console, It will show message successfully handled.
+5. Get the user from the postman by GET /users or /users/391a3656-9a01-47d4-8c6d-64b68c44f212.
+
+6. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"achievement","userId":"391a3656-9a01-47d4-8c6d-64b68c44f212","achievementsProviderId":"c77326d8-ef16-4be0-b844-d5c384b7bb8b","name":"achievement","uri":"https://google.com","certifierId":"b8726ca1-557e-4502-8f9b-25044b9c123d","certifiedDate":"2019-07-08T00:00:00.000Z"}}`
+7. Watch the app console, It will show message successfully handled.
+8. Get the achievement from the postman.
+
+9. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"achievementprovider","id":"c77326d8-ef16-4be0-b844-d5c384b7bb8b","name":"achievementprovider"}}`
+10. Watch the app console, It will show message successfully handled.
+11. Get the achievementprovider from the postman.
+
+12. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"attributegroup","id":"720c34f9-0fd4-46fd-9293-4a8cfdcd3e96","organizationId":"017733ad-4704-4c7e-ae60-36b3332731df","name":"attributegroup"}}`
+13. Watch the app console, It will show message successfully handled.
+14. Get the attributegroup from the postman.
+
+15. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"externalprofile","userId":"391a3656-9a01-47d4-8c6d-64b68c44f212","organizationId":"017733ad-4704-4c7e-ae60-36b3332731df","uri":"https:google.com"}}`
+16. Watch the app console, It will show message successfully handled.
+17. Get the externalprofile from the postman.
+
+18. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"organization","id":"017733ad-4704-4c7e-ae60-36b3332731df","name":"organization"}}`
+19. Watch the app console, It will show message successfully handled.
+20. Get the organization from the postman.
+
+21. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"role","id":"188446f1-02dc-4fc7-b74e-ab7ea3033a57","name":"role"}}`
+22. Watch the app console, It will show message successfully handled.
+23. Get the role from the postman.
+
+24. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"skill","id":"8a8c8d3a-9165-4dae-8a8c-f828cbe01d5d","skillProviderId":"63061b84-9784-4b71-b695-4a777eeb7601","externalId":"ba395d36-6ce8-4bd1-9d6c-754f0389abcb","uri":"https://google.com","name":"skill"}}`
+25. Watch the app console, It will show message successfully handled.
+26. Get the skill from the postman.
+
+27. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"skillprovider","id":"63061b84-9784-4b71-b695-4a777eeb7601","name":"skillprovider"}}`
+28. Watch the app console, It will show message successfully handled.
+29. Get the skillprovider from the postman.
+
+30. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"userattribute","userId":"391a3656-9a01-47d4-8c6d-64b68c44f212","attributeId":"b5a50f73-08e2-43d1-a78a-4652f15d950e","value":"userattribute"}}`
+31. Watch the app console, It will show message successfully handled.
+32. Get the userattribute from the postman.
+
+33. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"userrole","userId":"391a3656-9a01-47d4-8c6d-64b68c44f212","roleId":"188446f1-02dc-4fc7-b74e-ab7ea3033a57"}}`
+34. Watch the app console, It will show message successfully handled.
+35. Get the userrole from the postman.
+
+36. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"userskill","userId":"391a3656-9a01-47d4-8c6d-64b68c44f212","skillId":"8a8c8d3a-9165-4dae-8a8c-f828cbe01d5d","metricValue":"userskill","certifierId":"7cf786d9-a8c0-48ed-a7cc-09dcf91d904c","certifiedDate":"2019-07-08T00:00:00.000Z"}}`
+37. Watch the app console, It will show message successfully handled.
+38. Get the userskill from the postman.
+
+39. write message:
+  `{"topic":"u-bahn.action.create","originator":"u-bahn-api","timestamp":"2019-07-08T00:00:00.000Z","mime-type":"application/json","payload":{"resource":"attribute","id":"b5a50f73-08e2-43d1-a78a-4652f15d950e","name":"attribute", "attributeGroupId": "720c34f9-0fd4-46fd-9293-4a8cfdcd3e96"}}`
+40. Watch the app console, It will show message successfully handled.
+41. Get the attribute from the postman.
+
+42. These are full set of data for one user.
+Get users with enrich=true, verify all data above are returned
+
+43. Verify query parameters in each resource are works based on the swagger doc.
+When setting query parameter to the non-existing value, verify the app tries to get the data from QLDB in the console.
+
+44. By changing id values from above data, create another users or other resources.
+
+45. Test paging, query filtering with multiple resources.
+
+46. Please verify all specifications in the project.

config/default.js

@@ -5,13 +5,91 @@
 module.exports = {
   LOG_LEVEL: process.env.LOG_LEVEL || 'debug',
   PORT: process.env.PORT || 3001,
+
   AUTH_SECRET: process.env.AUTH_SECRET || 'CLIENT_SECRET',
   VALID_ISSUERS: process.env.VALID_ISSUERS ? process.env.VALID_ISSUERS.replace(/\\"/g, '')
     : '["https://topcoder-dev.auth0.com/", "https://api.topcoder.com"]',
-  API_VERSION: 'api/1.0',
 
-  AWS_KEY: process.env.AWS_KEY,
-  AWS_SECRET: process.env.AWS_SECRET,
+  PAGE_SIZE: process.env.PAGE_SIZE || 20,
+  API_VERSION: process.env.API_VERSION || 'api/1.0',
+
+  AWS_ACCESS_KEY_ID: process.env.AWS_ACCESS_KEY_ID,
+  AWS_SECRET_ACCESS_KEY: process.env.AWS_SECRET_ACCESS_KEY,
   AWS_REGION: process.env.AWS_REGION || 'us-east-1',
-  DATABASE: process.env.DATABASE || 'ubahn-db'
+  DATABASE: process.env.DATABASE || 'ubahn-db',
+
+  AUTH0_URL: process.env.AUTH0_URL,
+  AUTH0_AUDIENCE: process.env.AUTH0_AUDIENCE,
+  TOKEN_CACHE_TIME: process.env.TOKEN_CACHE_TIME,
+  AUTH0_CLIENT_ID: process.env.AUTH0_CLIENT_ID,
+  AUTH0_CLIENT_SECRET: process.env.AUTH0_CLIENT_SECRET,
+  AUTH0_PROXY_SERVER_URL: process.env.AUTH0_PROXY_SERVER_URL,
+
+  GROUP_API_URL: process.env.GROUP_API_URL || 'https://api.topcoder-dev.com/v5/groups',
+  BUSAPI_URL: process.env.BUSAPI_URL || 'https://api.topcoder-dev.com/v5',
+
+  KAFKA_ERROR_TOPIC: process.env.KAFKA_ERROR_TOPIC || 'common.error.reporting',
+  KAFKA_MESSAGE_ORIGINATOR: process.env.KAFKA_MESSAGE_ORIGINATOR || 'u-bahn-api',
+
+  // topics
+  UBAHN_CREATE_TOPIC: process.env.UBAHN_CREATE_TOPIC || 'u-bahn.action.create',
+  UBAHN_UPDATE_TOPIC: process.env.UBAHN_UPDATE_TOPIC || 'u-bahn.action.update',
+  UBAHN_DELETE_TOPIC: process.env.UBAHN_DELETE_TOPIC || 'u-bahn.action.delete',
+
+  // ElasticSearch
+  ES: {
+    HOST: process.env.ES_HOST || 'localhost:9200',
+    API_VERSION: process.env.ES_API_VERSION || '7.4',
+    // es mapping: _index, _type, _id
+    DOCUMENTS: {
+      achievementprovider: {
+        index: process.env.ACHIEVEMENT_PROVIDER_INDEX || 'achievement_provider',
+        type: '_doc'
+      },
+      attribute: {
+        index: process.env.ATTRIBUTE_INDEX || 'attribute',
+        type: '_doc'
+      },
+      attributegroup: {
+        index: process.env.ATTRIBUTE_GROUP_INDEX || 'attribute_group',
+        type: '_doc'
+      },
+      organization: {
+        index: process.env.ORGANIZATION_INDEX || 'organization',
+        type: '_doc'
+      },
+      role: {
+        index: process.env.ROLE_INDEX || 'role',
+        type: '_doc'
+      },
+      skill: {
+        index: process.env.SKILL_INDEX || 'skill',
+        type: '_doc'
+      },
+      skillprovider: {
+        index: process.env.SKILL_PROVIDER_INDEX || 'skill_provider',
+        type: '_doc'
+      },
+      user: {
+        index: process.env.USER_INDEX || 'user',
+        type: '_doc'
+      },
+      // sub resources under user
+      achievement: {
+        userField: process.env.USER_ACHIEVEMENT_PROPERTY_NAME || 'achievements'
+      },
+      externalprofile: {
+        userField: process.env.USER_EXTERNALPROFILE_PROPERTY_NAME || 'externalProfiles'
+      },
+      userattribute: {
+        userField: process.env.USER_ATTRIBUTE_PROPERTY_NAME || 'attributes'
+      },
+      userrole: {
+        userField: process.env.USER_ROLE_PROPERTY_NAME || 'roles'
+      },
+      userskill: {
+        userField: process.env.USER_SKILL_PROPERTY_NAME || 'skills'
+      }
+    }
+  }
 }

docker/Dockerfile

@@ -0,0 +1,12 @@
+# Use the base image with Node.js 12
+FROM node:12
+
+# Copy the current directory into the Docker image
+COPY . /ubahn_api
+
+# Set working directory for future use
+WORKDIR /ubahn_api
+
+# Install the dependencies from package.json
+RUN npm install
+CMD npm start

docker/docker-compose.yml

@@ -0,0 +1,11 @@
+version: '3'
+services:
+  ubahn_api:
+    image: ubahn_api:latest
+    build:
+      context: ../
+      dockerfile: docker/Dockerfile
+    env_file:
+      - api.env
+    ports:
+      - "3001:3001"

docker/sample.api.env

@@ -0,0 +1,9 @@
+AWS_ACCESS_KEY_ID=<AWS Access Key ID>
+AWS_SECRET_ACCESS_KEY=<AWS Secret Access Key>
+ES_HOST=<ES Host Endpoint>
+
+AUTH0_URL=<AUTH0 URL>
+AUTH0_AUDIENCE=<AUTH0 Audience>
+TOKEN_CACHE_TIME=500000
+AUTH0_CLIENT_ID=<AUTH0 Client ID>
+AUTH0_CLIENT_SECRET=<AUTH0 Client Secret>