This sample shows how to use DataRouter to upload data into Qrvey platform.
Qrvey DataRouter provides developers with a means for loading their data from a variety of different sources into Qrvey’s high performance, ElasticSearch backend. DataRouter provides an API that can be used for creating metadata data mapping and custom transformations, live update data push, and large volume historical data loads. It is designed to scale for all types of data volume and velocity scenarios.
Before using this sample, please have the following values
- Metadata URL. Contact help@qrvey.com if you don't have one.
- Postdata URL. Contact help@qrvey.com if you don't have one.
- Datarouter API Key
- Install NodeJS from https://nodejs.org (v12 recommended)
- Run the command to load data
Create the metadata specifying the fields and data types.
Example:
curl --location --request POST '{{metadataurl}}/v5/metadata?publicConnection=true' \
--header 'x-api-key: {{api-key}}' \
--header 'Content-Type: application/json' \
--data-raw '{
"MetaDataId": "quick_start_index_name",
"indexName": "quick_start_index_name",
"skipOnPartialData": false,
"dateFormat": "YYYY-MM-DDTHH:mm:ss",
"columnType": [
{
"columnName": "CompanyId",
"type": "numeric-general"
},
{
"columnName": "Company Name",
"type": "text-label"
},
{
"columnName": "Foundation Date",
"type": "date"
}
]
}'
In root folder with the following JSON:
{
"metadataId": "", //Use metadata created in the step #1
"postdataurl": "<postdataurl>", // Example: https://<API ID>.execute-api.<region>.amazonaws.com/Prod/dataload
"apikey": "<API key>"
}
Update the optionsAPI variable inside index.js file to execute the request and get the data to load on Qrvey.
const optionsAPI = {
hostname: 'openlibrary.org',
port: 443,
path: '/people/george08/lists.json',
headers: {
},
method: 'GET',
rejectUnauthorized: false
}
Load Data: node index.js