Collector for EVE Datasets

This script downloads subsets of the EVE (Events Visualization in Emergencies) feature table hosted on ArcGIS Online (AGOL). It creates 1 dataset with 1 global resource and 1 resource per country and will run every 14 days. DIEM account credentials are stored in a .env file in the root directory and are needed to run this script. The credentials are stored in LastPass.

Development

Environment

Development is currently done using Python 3.12. We recommend using a virtual environment such as venv:

    python -m venv venv
    source venv/bin/activate

In your virtual environment, install all packages for development by running:

    pip install -r requirements.txt

Installing and running

For the script to run, you will need to have a file called .hdx_configuration.yaml in your home directory containing your HDX key, e.g.:

hdx_key: "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
hdx_read_only: false
hdx_site: prod

You will also need to supply the universal .useragents.yaml file in your home directory as specified in the parameter user_agent_config_yaml passed to facade in run.py. The collector reads the key hdx-scraper-eve as specified in the parameter user_agent_lookup.

Alternatively, you can set up environment variables: USER_AGENT, HDX_KEY, HDX_SITE, EXTRA_PARAMS, TEMP_DIR, and LOG_FILE_ONLY.

To install and run, execute:

    pip install .
    python -m hdx.scraper.eve

Pre-commit

Be sure to install pre-commit, which is run every time you make a git commit:

    pip install pre-commit
    pre-commit install

With pre-commit, all code is formatted according to ruff guidelines.

To check if your changes pass pre-commit without committing, run:

    pre-commit run --all-files

Testing

Ensure you have the required packages to run the tests:

    pip install -r requirements-test.txt

To run the tests and view coverage, execute:

    pytest -c --cov hdx

Packages

uv is used for package management. If you’ve introduced a new package to the source code (i.e. anywhere in src/), please add it to the project.dependencies section of pyproject.toml with any known version constraints.

To add packages required only for testing, add them to the test section under [project.optional-dependencies].

Any changes to the dependencies will be automatically reflected in requirements.txt and requirements-test.txt with pre-commit, but you can re-generate the files without committing by executing:

    pre-commit run pip-compile --all-files

Project

Hatch is used for project management. The project can be built using:

    hatch build

Linting and syntax checking can be run with:

    hatch fmt --check

Tests can be executed using:

    hatch test