Skip to content

Question about pollutant units in the compact H5 dataset #3

Description

@Vinci-ui

Hi, I'm Vinci, thanks for the great benchmark and dataset! I have a question about the units of the 6 pollutants in the compact H5 files.

I understand from the README that metrics are computed in "original concentration units" after inverse normalization. However, when I checked the OpenAQ API parameter definitions, I noticed that each gaseous pollutant has two parameter entries — one in µg/m³ and one in ppm:

ID name units displayName
3 o3 µg/m³ O₃ mass
4 co µg/m³ CO mass
5 no2 µg/m³ NO₂ mass
6 so2 µg/m³ SO₂ mass
7 no2 ppm NO₂
8 co ppm CO
9 so2 ppm SO₂
10 o3 ppm O₃

Since the preprocessing script (convert_csv_to_h5.py) matches by parameter name rather than ID, both unit variants for the same pollutant (e.g., no2 from ID=5 and ID=7) appear to be merged into one channel. I also noticed the script only reads the value column, not unit, from the raw OpenAQ CSVs.

When I analyzed the data by geographic region, I found that US stations have CO values around ~0.25 while European stations have values around ~317, consistent with ppm vs µg/m³. This suggests the gaseous pollutant channels in the H5 files contain mixed units depending on the reporting country.

Could you confirm:

  1. Is this mixed-unit behavior expected / by design?
  2. For the scaler.csv statistics (e.g., CO mean = 228.46), were these computed on the raw mixed-unit values?
  3. Are there any plans to provide unit-normalized versions of the dataset?

Thanks for your time, and again for the excellent work on AirQualityBench!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions