Fetches data from the Eurostat API and persists it into an Elasticsearch database.
- Create the configuration file
.elastify-eurostatrc-exampleas a template.
- Add your Elasticsearch credentials to the configuration.
- Install Node dependencies with
$ npm install.
- Run the script with
$ npm start.
Prepare elasticsearch cluster
For elastic cloud: Create a cluster. Create roles and users to push data, and to read data Precreate the index with a mapping, else the year won't be a date.
node create_idx_mapping.js node push_data_citizens.js
Add Kibana to your cluster. Create index pattern. Create visualisations.
The scripts are helpers and have to be edited manually.
If you have to rerun:
- list indexes with
curl -u user:password https://host:port/_cat/indices
- delete your index with node
delete_idx.jsand start over.
If you want to see the data, no elasticsearch:
node log_citizen_data.js | less
to get all citizen codes (citizen.dic)
return 3706 codes