Hi, I'm Vinci, thanks for the great benchmark and dataset! I have a question about the units of the 6 pollutants in the compact H5 files.
I understand from the README that metrics are computed in "original concentration units" after inverse normalization. However, when I checked the OpenAQ API parameter definitions, I noticed that each gaseous pollutant has two parameter entries — one in µg/m³ and one in ppm:
| ID |
name |
units |
displayName |
| 3 |
o3 |
µg/m³ |
O₃ mass |
| 4 |
co |
µg/m³ |
CO mass |
| 5 |
no2 |
µg/m³ |
NO₂ mass |
| 6 |
so2 |
µg/m³ |
SO₂ mass |
| 7 |
no2 |
ppm |
NO₂ |
| 8 |
co |
ppm |
CO |
| 9 |
so2 |
ppm |
SO₂ |
| 10 |
o3 |
ppm |
O₃ |
Since the preprocessing script (convert_csv_to_h5.py) matches by parameter name rather than ID, both unit variants for the same pollutant (e.g., no2 from ID=5 and ID=7) appear to be merged into one channel. I also noticed the script only reads the value column, not unit, from the raw OpenAQ CSVs.
When I analyzed the data by geographic region, I found that US stations have CO values around ~0.25 while European stations have values around ~317, consistent with ppm vs µg/m³. This suggests the gaseous pollutant channels in the H5 files contain mixed units depending on the reporting country.
Could you confirm:
- Is this mixed-unit behavior expected / by design?
- For the
scaler.csv statistics (e.g., CO mean = 228.46), were these computed on the raw mixed-unit values?
- Are there any plans to provide unit-normalized versions of the dataset?
Thanks for your time, and again for the excellent work on AirQualityBench!
Hi, I'm Vinci, thanks for the great benchmark and dataset! I have a question about the units of the 6 pollutants in the compact H5 files.
I understand from the README that metrics are computed in "original concentration units" after inverse normalization. However, when I checked the OpenAQ API parameter definitions, I noticed that each gaseous pollutant has two parameter entries — one in µg/m³ and one in ppm:
Since the preprocessing script (
convert_csv_to_h5.py) matches by parameter name rather than ID, both unit variants for the same pollutant (e.g.,no2from ID=5 and ID=7) appear to be merged into one channel. I also noticed the script only reads thevaluecolumn, notunit, from the raw OpenAQ CSVs.When I analyzed the data by geographic region, I found that US stations have CO values around ~0.25 while European stations have values around ~317, consistent with ppm vs µg/m³. This suggests the gaseous pollutant channels in the H5 files contain mixed units depending on the reporting country.
Could you confirm:
scaler.csvstatistics (e.g., CO mean = 228.46), were these computed on the raw mixed-unit values?Thanks for your time, and again for the excellent work on AirQualityBench!