Data Sets

Mat Schaffer edited this page Dec 12, 2018 · 15 revisions

Radiation data (via api.safecast.org)

All of the Bgeigie log files are uploaded to api.safecast.org. Additional pointcast fixed radiation sensors publish data into this database.

REST API

This service has a public rest API. Examples are available at https://api.safecast.org/home and Swagger definition is available at https://api.safecast.org/api_docs.json

Daily export

Current Safecast Radiation Dataset

Complete Radiation dataset (zipped), updated daily. This is a large download.

Air and Radiation data (via ingest.safecast.org)

The ingest.safecast.org endpoint was developed for the new Solarcast project to support a much more flexible data model. The data stores for this app contain all the Solarcast data as well as data from pointcast and various fixed sensors.

Raw S3 export

Data is exported every hour to s3 available via standard S3 tools. For example with the aws cli:

❯ aws s3 ls s3://safecastdata-us-west-2/ingest/prd/s3raw/2017-10-01/

The format is in lines of JSON, delimited by two line feeds. The JSON objects are wrapped first with a versioning envelope, then the payload key in that envelope which is the payload that was posted to ingest.safecast.org.

Example:

❯ aws s3 cp s3://safecastdata-us-west-2/ingest/prd/s3raw/2017-10-01/00/12 - | head -n4
{"version":1,"payload":{"device":10004,"when_captured":"2017-09-30T23:12:38Z","loc_lat":34.4883,"loc_lon":136.165,"loc_alt":535,"loc_olc":"8Q6RF5Q7+8XH","lnd_7318u":2100,"dev_test":true,"service_uploaded":"2017-09-30T23:12:38Z","service_transport":"pointcast:210.142.99.156","service_md5":"f0b668c27bcd19ad316cfff6308918b6","service_handler":"i-0c65ac97805549e0d"}}

{"version":1,"payload":{"device":178,"when_captured":"2017-09-30T23:12:54Z","loc_lat":37.05676,"loc_lon":-122.138336,"loc_olc":"849V3V46+PM3","lnd_7318u":46,"service_uploaded":"2017-09-30T23:12:54Z","service_transport":"ngeigie:107.161.164.166","service_md5":"94a54e3e2fc510030bc2d3a38b4206c2","service_handler":"i-051a2a353509414f0"}}

Each device is permitted to publish their own fields but here are some of the fields expected from our Solarcast device fleet:

  • when_captured: Device timestamp when measurement was taken
  • service_uploaded: Device timestamp when measurement was uploaded
  • loc_lat: Device latitude
  • loc_lon: Device longitude
  • opc_pm01_0: Alphasense OPC-N2 air sensor, PM 1.0 (units: μg/m³)
  • opc_pm02_5: Alphasense OPC-N2 air sensor, PM 2.5 (units: μg/m³)
  • opc_pm10_0: Alphasense OPC-N2 air sensor, PM 10.0 (units: μg/m³)
  • pms_pm01_0: Plantower PMS5003 air sensor, PM 1.0 (units: μg/m³)
  • pms_pm02_5: Plantower PMS5003 air sensor, PM 2.5 (units: μg/m³)
  • pms_pm10_0: Plantower PMS5003 air sensor, PM 10.0 (units: μg/m³)
  • pms2_pm01_0: Plantower PMS5003 air sensor (secondary), PM 1.0 (units: μg/m³)
  • pms2_pm02_5: Plantower PMS5003 air sensor (secondary), PM 2.5 (units: μg/m³)
  • pms2_pm10_0: Plantower PMS5003 air sensor (secondary), PM 10.0 (units: μg/m³)
  • lnd_712u: Unshielded LND 712 radiation sensor (units: μSv/h)
  • lnd_712c: Shielded LND 712 radiation sensor (units: μSv/h)
  • lnd_7318u: Unshielded LND 7318 radiation sensor (units: μSv/h)
  • lnd_7318c: Shielded LND 7318 radiation sensor (units: μSv/h)
  • lnd_7128ec: Energy-compensated LND 7128 radiation sensor (units: μSv/h)
  • lnd_78017u: Unshielded LND 78017 radiation sensor (units: μSv/h)
  • lnd_78017c: Shielded LND 78017 radiation sensor (units: μSv/h)
  • lnd_78017w: Water-attenuated LND 78017 radiation sensor (units: μSv/h)
  • env_temp: External temperature (units: °C)
  • env_humid: External humidity (units: RH%)
  • env_press: External air pressure (units: hPa)

Kibana & Elasticsearch access

Data that comes into ingest.safecast.org is also available in kibana & elasticsearch at:

This system requires an account to access but anyone is welcome. Please email mat@schaffer.me to request an account.

Grafana access

Data that comes into ingest.safecast.org is also available in grafana as well.

See https://grafana.safecast.cc/dashboard/db/recent-data for an example.

This is publicly browsable, or if you're added to the Safecast github organization you can authenticate with your github account to be given write access.

Postgres access

We also have a publicly accessible postgres replica available at ingest-replica1.prd.safecast.cc

This system requires an account to access but anyone is welcome. Please email mat@schaffer.me to request an account.

SNS push

All ingest data also feeds through the AWS SNS topic arn:aws:sns:us-west-2:985752656544:ingest-measurements-prd.

If you have an endpoint you'd like to subscribe to our topic for realtime updates, we can assist with that. Please email mat@schaffer.me with information about the endpoint you would like to subscribe.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.