Skip to content

Repository files navigation

earthlens — 48 Earth-observation providers · one facade · modular install

Tests Wheel Docs PyPI - Python Version License: GPL v3 pre-commit conda-forge feedstock

codecov GitHub last commit GitHub forks GitHub Repo stars

Current release info

Name Downloads Version Platforms
Conda Recipe Conda Downloads Downloads Downloads Downloads PyPI - Downloads GitHub all releases Conda Version PyPI version Anaconda-Server Badge Conda Platforms

earthlens — a unified Python client for satellite & climate data

earthlens gives you one consistent Python API for downloading satellite, climate, and geospatial data from 48 providers — climate reanalysis, satellite imagery, ocean models, weather forecasts, natural-hazard feeds, air quality, biodiversity, population, and more — and turning the results into analysis-ready GeoTIFFs, GeoDataFrames, or tables.

It is part of the serapeum-org open-source ecosystem and is built on top of pyramids-gis for raster I/O.

Why earthlens?

A satellite orbiting the earthlens globe

Every provider speaks its own dialect: CHIRPS is anonymous FTP with date-coded filenames, ERA5-on-S3 is unsigned object storage with a per-month layout, the ECMWF CDS expects a JSON request body validated against a constraints graph, Google Earth Engine is a server-side image-collection model, and the other 44 each have their own. earthlens flattens all of it into one call:

from earthlens.core import EarthLens

earthlens = EarthLens(
    data_source="ecmwf",          # or "chc" (alias "chirps"), "amazon-s3", "gee"
    temporal_resolution="monthly",
    start="2022-01-01",
    end="2022-12-01",
    variables={
        "reanalysis-era5-single-levels-monthly-means": [
            "2m-temperature",
            "total-precipitation",
        ],
    },
    lat_lim=[37.0, 38.0],
    lon_lim=[23.0, 24.0],
    path="data/era5",
)
earthlens.download()

You get back per-date, per-variable GeoTIFFs in data/era5/ — clipped to your bbox, ready to feed into a hydrology model, a PV-yield notebook, a heat-wave study, or anything else downstream.

Features

  • 48 backends, one facade. EarthLens(data_source=...) routes to any provider without changing the rest of your code. Backends are discovered through entry points and imported lazily, so the SDK for a provider you never touch is never loaded.
  • Cross-provider discovery. find("precipitation") tells you which of the 48 providers serve a dataset, offline, before you commit to one; search(...) dry-runs a request and lists exactly what it would fetch.
  • YAML variable catalogs per provider — every variable carries metadata: NetCDF name, units, accumulation semantics (is_flux), allowed pressure levels, monthly counterparts. Browseable with Catalog().get_variable(...).
  • Pre-flight request validation against the live CDS constraints.json graph. Bad date / area / variable combinations are rejected before bytes go over the wire, with actionable error messages.
  • Temporal aggregation built in. Pass an AggregationConfig to download() and earthlens emits aggregated GeoTIFFs alongside the raw NetCDFs. op="auto" reduces state variables (temperature, SST, soil moisture) by mean and flux variables (precipitation, radiation, evaporation) by sum — the physically correct choice driven by catalog metadata.
  • Pressure-level support. ERA5 pressure-level fields (4-D NetCDFs) can be sliced to a specific level on download.
  • Bbox cropping & NetCDF→GeoTIFF conversion are handled by pyramids-gis under the hood.
  • Modular install extras — only install the SDK for the backend you need (pip install earthlens[ecmwf], [s3], [gee]).
  • Bounded by design. Streamed, atomic downloads; pooled HTTP connections with Retry-After-aware back-off; an exact limit= cap for vector/tabular requests; re-runs skip artefacts that are already complete.
  • Strictly typed. Pydantic v2 models for catalog rows and request specs; modern PEP 585/604 type hints; Python 3.11 – 3.14 tested in CI.

Supported data sources

earthlens wraps 48 providers behind the one EarthLens(data_source=..., ...) facade — pass the data_source value below and everything else (auth, request shaping, output format) is handled per-backend. See Data Sources for the full walkthrough of each one.

Air quality

Provider data_source
AirNow (US EPA) airnow
European Environment Agency eea-aq
OpenAQ openaq
Sensor.Community sensor-community

Biodiversity & protected areas

Provider data_source
GBIF gbif
IUCN Red List iucn
OBIS obis
Protected Planet (UNEP-WCMC) wdpa

Climate reanalysis & projections

Provider data_source
Copernicus Climate Data Store (ECMWF) ecmwf
NOAA Physical Sciences Laboratory climate-indices
WCRP CMIP6 cmip6

Disasters & risk

Provider data_source
FDSN fdsn
GDACS gdacs
NASA FIRMS firms
ThinkHazard! (GFDRR/World Bank) risk-indicators

Elevation & bathymetry

Provider data_source
Copernicus DEM (ESA) dem
GEBCO bathymetry

Glaciers & cryosphere

Provider data_source
NSIDC Randolph Glacier Inventory glaciers

Humanitarian data

Provider data_source
Humanitarian Data Exchange (UN OCHA) hdx

Hydrology

Provider data_source
NOAA National Water Model nwm
USGS National Water Information System usgs-water

Multi-mission imagery & data platforms

Provider data_source
AWS Open Data amazon-s3
EUMETSAT eumetsat
Google Earth Engine gee
JAXA jaxa
NASA Earthdata earthdata
NOAA GOES-R goes
STAC (SpatioTemporal Asset Catalog) stac
Sentinel Hub sentinel-hub
openEO openeo

Ocean

Provider data_source
Argo Program argo
Copernicus Marine Service cmems
NOAA ERDDAP erddap

Population & human settlement

Provider data_source
European Commission Joint Research Centre (GHSL) ghsl
WorldPop worldpop

Precipitation & drought

Provider data_source
Climate Hazards Center (UCSB) chc
Copernicus European Drought Observatory / NDMC drought

Renewable energy

Provider data_source
Global Solar Atlas / Global Wind Atlas (World Bank/ESMAP) solar-wind-atlas
National Laboratory of the Rockies (formerly NREL) nrel
PVGIS (EU JRC) pvgis

SAR / radar imagery

Provider data_source
Alaska Satellite Facility (ASF) asf

Soil

Provider data_source
ISRIC SoilGrids soilgrids

Tropical cyclones

Provider data_source
Tropycal tropycal

Vector basemaps & boundaries

Provider data_source
OpenStreetMap osm
Overture Maps Foundation overture
geoBoundaries admin

Weather forecast (NWP)

Provider data_source
Herbie (NWP archive access) nwp

Weather radar

Provider data_source
NOAA NEXRAD radar

Logos are each provider's own mark, used only to identify which service a backend talks to (not an endorsement of earthlens by that provider) — see docs/_images/logos/ATTRIBUTION.md for sourcing and rights notes on every logo, including the handful of providers with no distinct mark of their own.

Installation

earthlens is published on conda-forge and PyPI.

# pip — latest release
pip install earthlens

# conda
conda install -c conda-forge earthlens

# pip — bleeding edge
pip install git+https://github.com/serapeum-org/earthlens

To list all available versions on your platform:

conda search earthlens --channel conda-forge

A plain pip install is enough — earthlens pulls in everything it needs.

Backend SDKs are optional and pulled in by extras:

pip install earthlens[ecmwf]   # cdsapi
pip install earthlens[s3]      # boto3 + botocore
pip install earthlens[gee]     # earthengine-api
pip install earthlens[all]     # every backend SDK

[all] deliberately omits exactly two extras — argo and osm-pbf. argopy requires xarray>=2025.7 while openeo (which is in all) caps xarray<2025.1.2, and pyrosm pulls the sdist-only cykhash, which would make [all] need a C compiler. osm itself is included. Both excluded extras install fine on their own — see what [all] excludes.

For a development environment the repo is a uv workspace — dev and docs are dependency groups, not extras:

uv sync --extra all --group dev

See Contributing for the full setup.

Quick examples per backend

CHIRPS daily rainfall — anonymous FTP, no credentials.

from earthlens.core import EarthLens

EarthLens(
    data_source="chc",
    temporal_resolution="daily",
    start="2009-01-01",
    end="2009-01-10",
    variables=["precipitation"],
    lat_lim=[4.19, 4.64],
    lon_lim=[-75.65, -74.73],
    path="data/chirps",
).download(cores=4)  # parallel FTP fetch

ERA5 monthly via AWS public S3 — unsigned, fast, no API key.

EarthLens(
    data_source="amazon-s3",
    temporal_resolution="monthly",
    start="2020-01-01",
    end="2020-12-01",
    variables=["air_temperature_at_2_metres", "precipitation_amount_1hour_Accumulation"],
    lat_lim=[30.0, 35.0],
    lon_lim=[28.0, 35.0],
    path="data/era5-s3",
).download()

ECMWF CDS with on-the-fly aggregation. Downloads daily ERA5, then writes monthly GeoTIFFs aggregated with the right reduction per variable (mean for temperature, sum for precipitation):

from earthlens.core import EarthLens, AggregationConfig

EarthLens(
    data_source="ecmwf",
    temporal_resolution="daily",
    start="2022-06-01",
    end="2022-08-31",
    variables={
        "reanalysis-era5-single-levels": [
            "2m-temperature",
            "total-precipitation",
        ],
    },
    lat_lim=[37.0, 38.0],
    lon_lim=[23.0, 24.0],
    path="data/athens-summer",
).download(aggregate=AggregationConfig(freq="1MS", op="auto"))

Google Earth Engine — server-side collection, downloaded as GeoTIFFs. The request is {asset_id: [band, ...]}, and GEE needs a service account:

EarthLens(
    data_source="gee",
    temporal_resolution="monthly",       # one composite image per month
    start="2020-06-01",
    end="2020-08-31",
    variables={"UCSB-CHG/CHIRPS/DAILY": ["precipitation"]},
    lat_lim=[28.0, 32.0],
    lon_lim=[30.0, 34.0],
    path="data/gee",
    scale=5566,                          # output pixel size in metres
).authenticate(
    service_account="my-sa@my-project.iam.gserviceaccount.com",
    service_key="/path/to/key.json",
).download()

Aggregation: state vs flux

ERA5 mixes two physically distinct kinds of variables:

  • State variables are instantaneous samples — temperature, SST, soil moisture, snow depth. Aggregating in time means averaging.
  • Flux variables are accumulated over each timestep — precipitation, radiation, evaporation, surface heat fluxes. Aggregating in time means summing.

Mixing those up produces silently wrong results (a "monthly mean" of precipitation under-reports rainfall by ~30×). earthlens's catalog tags every variable with is_flux, and op="auto" reads that flag to pick the right reduction:

from earthlens.ecmwf import Catalog
spec = Catalog().get_variable(
    "reanalysis-era5-single-levels", "total-precipitation"
)
print(spec.is_flux)  # True  -> auto-aggregate by SUM

You can override with op="mean" | "sum" | "max" | "min" when you know better than the catalog.

Authentication

Roughly half the backends need no credentials at all. Common ones:

Source What you need
CHIRPS / CHC Nothing — anonymous FTP.
Amazon S3, Copernicus DEM, GOES, NWM, NEXRAD Nothing — unsigned, public buckets.
GDACS, GHSL, Overture, HDX, SoilGrids, PVGIS, admin Nothing — public HTTP.
ECMWF / CDS A free CDS account and a ~/.cdsapirc with your API key.
GEE A Google Earth Engine project and a service-account JSON key.
CMEMS, Earthdata, ASF, EUMETSAT, Sentinel Hub, openEO A provider login.
OpenAQ, AirNow, FIRMS, WDPA, IUCN, NREL, GFW A free API key or token.

Where credentials go is backend-specific: some take them as constructor keywords (CMEMS's service_username= / service_password=), others in authenticate(...) (GEE's service_account=, FIRMS's api_key=), and most fall back to an environment variable. Each backend's page says which. The full per-provider matrix is in Supported providers.

Documentation

Full docs, API reference, architecture diagrams, and a gallery of domain-specific example notebooks (hydrology, oceanography, agriculture, solar/wind energy, heat waves, drought, snow & cryosphere, climate-change anomalies) live at:

https://serapeum-org.github.io/earthlens/

Start here:

Page What it covers
Getting started Install to first file on disk.
Discovering datasets sources() / find() / search() across all 48 providers.
Supported providers Keys, output kinds, auth, and extras for every backend.
Temporal aggregation Reduce a stack into windowed composites.
Troubleshooting When a download fails, and what to change.
Migration guide Breaking changes by release.
Architecture How the facade, registry, and backends fit together.

Contributing

Issues, PRs, and discussions are welcome on GitHub. The repo uses pre-commit with ruff (ruff-check + ruff-format), mypy, and bandit — install the hooks once with pre-commit install.

See the contributing guide for the workspace layout, how to run the tests, and how to add a new provider backend.

License

GPL v3. See LICENSE.

Releases

Packages

Used by

Contributors

Languages