A Krawler based service to download atmospheric data from the French gelectrical network operated by RTE.
The k-icos job allow to scrape atmospheric data from the [RTE API]. The downloaded data are stored within a MongoDB database and more precisely in 2 collections:
- the
icos-observations
collection stores the observation data - the
icos-stations
collection stores the station positions
All records are stored in GeoJson format.
The project consists in 2 jobs:
- the
stations
job scrapes the available stations according a specific cron expression. By default, every day at midnight. - the
observations
job scrapes the observations data according a specific cron expression. By default every hour.
Variable | Description |
---|---|
DB_URL |
The database URL. The default value is mongodb://127.0.0.1:27017/icos |
OBJECT_SPEC_FILTER |
A filter used to select only latest data objects containing this pattern. The default value is radon data , could be eg CO data , CO2 data , CH4 data |
HISTORY |
The duration of the observations data history the job has to download. It must be expressed in seconds and the default value is 86 400 000 (1 day) |
DEBUG |
Enables debug output. Set it to krawler* to enable full output. By default it is undefined. |
Variable | Description |
---|---|
DB_URL |
The database URL. The default value is mongodb://127.0.0.1:27017/icos |
TTL |
The observations data time to live. It must be expressed in seconds and the default value is 604 800 (7 days) |
OBJECT_SPEC_FILTER |
A filter used to select only latest data objects containing this pattern. The default value is radon data , could be eg CO data , CO2 data , CH4 data |
OBJECT_VARIABLE |
The name of the variable contained in the corresponding data objects. The default value is rn , could be eg co , co2 , ch4 |
HISTORY |
The duration of the observations data history the job has to download. It must be expressed in seconds and the default value is 86 400 000 (1 day) |
DEBUG |
Enables debug output. Set it to krawler* to enable full output. By default it is undefined. |
We personally use Kargo to deploy the service.
Please refer to contribution section for more details.
This project is sponsored by
This project is licensed under the MIT License - see the license file for details