Repository navigation
Releases
Releases · jmineau/uataq
v2026.11.1
Added
uataq.gsv:gsv.get_obsreads the Google Street View cars' Salt Lake observations (May 2019 to March 2020) as one time-sorted table with positions, for chosen pollutants, cars and a time range. The QC flags are kept, andgood_only=Truekeeps only rows with a good position and empties values whose flag is not 0. It reads the Lin group's merged table on CHPC or the published copy (doi:10.5281/zenodo.7605718). A module of its own likeuataq.sodar, not a site of the network. (#195, #196)- TRX02's temperature, humidity and pressure sensors (2016-02-04 to 2017-06-20) are readable through a new
Metinstrument. (#160, #193) get_obs(group="horel")reads TRX02's LGR NO2 from the Horel group's pilot archive (2016-06-27 to 2017-06-20), where each row is the mean of the minute before its time. The default group is stilllin, so existing calls do not change. (#160, #194)UATAQ_HOREL_DIRpoints the package at the Horel group's archive root. It is read each time a path is needed, so setting it afterimport uataqtakes effect, and a value that is not a directory raisesFileNotFoundErrornaming the variable. (#110, #192)
v2026.11.0
Added
Site.get_obsandMobileSite.get_obslabel each row with its instrument in aninstrumentcolumn after the measurement columns, built the same way for both. The column holds text, so a plain.mean()on the table now needsnumeric_only=True(or the measurement columns picked first). Row counts and values are unchanged. (#163)
Changed
- Breaking:
Network.siteslists only the sites the network kept, in the order given, not every name that was asked for (a misspelled site or one that does not measure the pollutant is no longer in it). The "none of the sites given" error still names the sites the caller gave. (#165, #167) - Breaking: a site ID given as a string is one site:
Network("WBB", "CO2")andget_network_obs("TRX01", "CO2")are the network of that site, and so isget_availability(lvls="final"). A string used to be iterated as letters and never made a network."all"is unchanged. (#174) - Data change:
read_data(..., "lgr_ugga", lvl="raw")before 2015-12-29 returns UTC times. Those raw files were logged in Mountain time, so times move later by 6 h in summer and 7 h in winter, as the pipeline'slgr_ugga_init.rdoes, and the raw rows of 2015-12-29 (89,144 rows at the five sites) are left out. A raw read that starts before 2015-12-30 opens files from two days before its start. (#175, #178) - The
teledyne_t500uqaqc level names its sample temperatureInternal_T_C, as its raw level and the T300 and T400 do, where it came out asSamp_T_C. The values are the same. (#185, #188) - WBB is named the William Browning Building. (#172)
Fixed
- A horel GPS read at
finalkeepsAltitude_msl, equal row for row to the qaqc read. (#176, #177) - The horel
finallevel drops an ES642 row that has an ambient temperature or pressure but no PM2.5. Every other horel final read is the same. (#186, #189) - The ES405 values that horel's qaqc and final files name for the ES642 at BUS01 and BUS02 are read. (#171)
- The first row of an air-trend raw file that has no header line is kept. (#187)
- Removal dates are corrected where the last reading shows a later one: BUS03's
2b_405(#156), TRX01'slgr_ugga(#162) and FRU'slgr_ugga(#170). TRX01'smetone_es642is installed on 2014-12-05, the day of the field team's note. (#169)
v2026.10.2
Added
LINGROUP_MEASUREMENTS_DIRsets the Lin group's measurements root when it is
set and not empty. It is read each time a path is needed, so setting it after
import uataqworks, except for the pipeline configuration files, which are
read at import.groupspaces.lin.MEASUREMENTS_DIR,CONFIG_DIRandDATA_DIR
keep their names and follow the variable. A value that is not a directory
raisesFileNotFoundErrornaming the variable.MobileSite.get_obshas aninstrumentcolumn naming the instrument that
logged each row, asget_network_obsalready has. At TRX01 the rows of
lgr_uggaandlgr_ugga_manual_calshare columns, so this is the only way to
tell them apart. The column follows the measurement columns and comes before
the GPS ones, in wide and long format.MobileSite.get_obswarnsEmptyResultWarningwhen it read rows but none
found a GPS fix (TRX01's methane in 2018-09 and 2018-10, its lin ozone in
2017-08), says how many rows were read and points toinclude_gps=False. It
returned an empty table without a word.
Changed
- The default Lin group measurements root is the group's dedicated measurements
space, not a link to it in another group space. The data read are the same.
LinGroup.data_diris now a property. - uataq declares minimum versions: geopandas 1.1, numpy 1.24, pandas 2.0 and
xarray 2022.3. geopandas 1.0 and older importshapely.geos, which warns on
shapely 2.1. - A bad
TimeRange(TimeRange("2024", start="2023-01-01")), a bad data
level inInstrument.read_data, and the raw-only check in
LGR_UGGA_File.get_filesraiseValueErrorwith the same messages. They were asserts, so under
python -Othe check was skipped andstartwas dropped without a word. - Docs: the front page cites Mendoza et al. (2024), the electric bus paper, for
the bus data, and no longer cites Mitchell et al. (2018, PNAS), the long-term
CO2 record. From a new student's read: the README's installation link works,
the pin example names the newest release, wide tables say they are shortened,
the loop that skips sites with no data shows its log lines, docstrings say
time_rangeis UTC and whatread_dataandNetwork.siteshold, the SODAR
example uses a month USDR1 has data for,num_processesis said to speed up
final-level reads too, and large reads, the Slurm script, the dev setup, the
horel clock and the raw-only months each have one page.
Removed
uataq.filesystem.groupspaces.lin.dms2dd. Nothing called it, and
it was wrong for negative degrees (dms2dd(-111, 53, 24)gave -110.11, not
-111.89);lair.geo.dms2ddis the one to use.
Fixed
- Rows that share a timestamp keep their file order in every session. A read
sorts its rows stably, and so does the list of files in one period. The
default sort could swap them, as it can for TRX01'slgr_ugga_manual_cal
qaqc files, which repeat about 1,400 timestamps a month. - A row with no valid GPS time of day gets a NaT with the
nsunit. The unitless
NaT warns on NumPy 2.5 and will become an error in a future NumPy. Site.get_obsgives each pollutant only its own columns."NO"also
returnedNO2_ppbandNOx_ppb, and"PM1"returnedPM10_ugm3; in the long
format their values were rows of the same table. WBB'steom_1400ab
PM2.5_ugm3_30minandPM2.5_ugm3_1hr(2016-09 to 2017-03) are no longer in
get_obs("PM2.5").- Raw LGR files with other column layouts are read by their header. Ten files
(ROO and HPL, 2015-11) have 77 columns and gave no rows: every name landed on
the wrong field. A layout that is neither 23 nor 24 columns takes the 21
measurement columns,Fit_Flagby name and the MIU description; one without a
Fit_FlagraisesParserError. - A raw read of the horel pilot archive keeps only the rows before the pilot
phase ends. In 2018-11 TRX01's and TRX02's pilot files run on with a
once-a-minute copy of the main archive's rows, so they came back twice
(TRX012b_205: 14,685 of 521,690 rows). uataq.gpshaversine distances are never NaN. Rounding could put the
haversine term just above 1 for near-antipodal points.
v2026.10.1
Added
- Raw horel reads take the
*.h5_prior405upgradefile of a month beside its
.h5. horel split the raw files of the month of each MetOne ES642-to-ES405
swap (BUS02 2025-03, BUS03 2024-12, TRX01, TRX02 and TRX03 2025-01): the rows
before the swap are in the second file, which uataq did not read. The
ES642's last 3 to 13 days before each swap can now be read (raw only; horel's
qaqc and final files do not have them), and so can TRX01-03's raw ozone
(2b_205) and GPS from 1 January 2025 to the swap, which locate those rows.
The ES405 does not read these files.
Changed
- Docs: rewritten for a new student and organized by task, with no redirects
from the old addresses. The navbar has four sections, Getting started,
Guides, Reference and Contributing, each a folder of pages, and the front page
says what UATAQ is, shows a first read and how to cite the data. Where the
2026.10.0 pages went: "Quick Start" is "Read data" (guides/read_data.html),
rewritten; "General" is "Input Parameters"
(reference/input_parameters.html), rewritten, withformatandrecent;
"Installation" isgetting_started/installation.html, covers Jupyter on Open
OnDemand, and its development setup is on Contributing; "Configuration" is
reference/config.htmland no longer printsconfig.json; "Pollutants" is in
"Columns" (reference/columns.html); the module pages ("Laboratory",
"Filesystem", "Sites", "Instruments", "GPS", "SODAR") are in the API
reference. New pages: "Your first data" (a month of CO2 and CH4 in local time,
hourly means, a daily-cycle plot, and what to write down), "Find sites and
data" (which sites measure what, when they have data, the newest data), "Data
levels and flags" (the levels,ID, whatfinalkeeps, the flags, empty
_cal), "Mobile sites" (positions, the two groups' clocks, TRX01's two LGR
instrument names, the months only atraw), "When there is nothing to read",
"Sites and instruments" (tables made from the configuration) and "Columns"
(the columns of each instrument and level). Each fact is on one page, and the
others link to it. - Docs: the API reference is generated, with a page for each module, class,
function and class member, as in the docs of the other packages (the page
"Functions and modules",reference/functions_and_modules.html, lists them,
the public functions first and the internals last).
gps.html,instruments.html,pollutants.html,sodar.htmland
filesystem.htmlno longer exist: the module pages are
api/uataq.gps.html,api/uataq.instruments.html,
api/uataq.pollutants.html,api/uataq.sodar.htmland
api/uataq.filesystem.html, and each class and function is underapi/
(api/uataq.instruments.GPS.html). What the GPS, Pollutants and File System
pages said by hand is the docstring of the module, shown at the top of its
page. The File System page'sDataFile,GroupSpaceand functions are now
underuataq.filesystem.core, and the group spaces' pages under
api/uataq.filesystem.groupspaces.horel.htmland.lin.htmlhave one page
for each class.read_dataandget_obsmoved from "Input Parameters"
(input_parameters.html#uataq.read_dataand#uataq.get_obs), and
get_recent_obs,get_network_obsandNetworkfrom the closing
"Reference" section of "What data exists" (what_data_exists.html#reference,
#uataq.get_recent_obs,#uataq.get_network_obsand#uataq.Network), to
api/uataq.read_data.html,api/uataq.get_obs.htmland so on.TimeRange
(api/uataq.timerange.html) and the exceptions and warning
(api/uataq.errors.html), which had no page, are documented. - Docs: the "Laboratory" and "Sites" pages are generated like the rest of the
API reference, solaboratory.htmlandsites.htmlno longer exist.
laboratory,get_siteand theLaboratoryclass have pages underapi/
(api/uataq.laboratory.html,api/uataq.get_site.htmland
api/uataq.Laboratory.html; the class is documented asuataq.Laboratory,
no longer asuataq._laboratory.Laboratory), andSiteandMobileSiteare
listed on the module pageapi/uataq.sites.html, each with its own page
(api/uataq.sites.Site.html). The anchorslaboratory.html#uataq.laboratory,
#uataq.get_site,#uataq._laboratory.Laboratory,
sites.html#uataq.sites.Siteand#uataq.sites.MobileSitemoved to those
pages.uataq.laboratoryhides its value, which would be the whole
configuration.Site.pollutant_instrumentshas a docstring. - Docs: the levels are called "data levels" on every page and docstring.
"Input Parameters" retitles its "Processing Level" section "Data Level" (the
anchorinput_parameters.html#processing-levelis now#data-level).
"Reading data" says the mobile sites are on TRAX trains and buses, as the
front page does. Instrument.model,SensorMixin.pollutants,Sodar.modeland
Sodar.species_measuredhave docstrings, and the Attributes of
LinDatFileandLGR_UGGA_Filename the ones their__init__sets
(instrumentandlvl;meta).- Configuration: HDP, HEB and SUG were taken down, so they are marked
is_active: falseand their instruments now have removal dates (HDP
lgr_ugga2025-02-08, HEBlicor_62622025-05-30, SUGlicor_7000
2025-08-23). Each is the midnight after the last sample in the archive,
except HEB'slicor_6262, which is the day HEB switched to alicor_7000.
A read after one now reports the instrument as inactive instead of finding
no files. HEB'slicor_7000, which replaced itslicor_6262from
2025-05-30 to 2025-09-16, is added. - Configuration: WBB's
magee_ae33and TRX02'slgr_no2now have removal
dates, 2024-06-20 and 2017-06-21, the midnight after their last samples
(2024-06-19 17:41:53 and 2017-06-20 05:28:52 UTC, at raw, qaqc and final).
No row is read differently: neither has data after its last sample. A
get_obsof WBB's BC, or of TRX02's NO2 before itsteledyne_t500uwas
installed (2020-12-22), for a range wholly after the date now raises
ReaderErrorsaying the instrument is inactive, instead of returning an
empty frame. - The error for a read that finds no data at a site says what you asked for and
which files exist.Site.get_obs("CO2", time_range="2021-08")at WBB, which
has no final file that month, raisedNo data found for ['lgr_ugga'] at WBB (lgr_ugga in lin groupspace).It now raisesNo final data found for CO2 at WBB: lgr_ugga in lin groupspace: No files found within the specified time range (files run from 2014-02 to 2026-10, with gaps).The first and last
period come from the files the read had already listed, so the error costs
no extra read.read_datanames
its instruments whereget_obsnames the pollutants, a read with nolvl
leaves the level out, and the warning for each instrument andNetwork's
warning for a site carry the same periods.ReaderError.detailholds what
follows the colon (#63). - A read with files for the range but no rows in them now warns.
get_obs
of WBB's CO2 for 2024-03-04, a day with no rows in the March 2024 file,
returned an empty table and said nothing. It still returns the empty table,
and now warnsNo final data found for CO2 at WBB (2024-03-04 00:00:00 to 2024-03-05 00:00:00): the files for lgr_ugga exist but hold no data in that range.get_recent_obsandNetwork.get_obs(and soget_network_obs)
warn the same way when their table has no rows. It is a Python warning,
uataq.errors.EmptyResultWarning(aUserWarning), so it shows without
turning the log on. A table with rows from any instrument or site does not
warn, andread_datais unchanged (#63). - A read of several pollutants now warns when it leaves one out.
get_obs(["CO2", "O3"], time_range="2021-08")at WBB, which has no final CO2 file that month,
returned only the O3 column and said nothing. It still returns that table,
and now warns
No final data found for CO2 at WBB (2021-08-01 00:00:00 to 2021-09-01 00:00:00): lgr_ugga gave none, so the table leaves out CO2.The warning names
every pollutant left out and the instruments that gave none, whether they had
no files for the range or were not installed then. It is a Python warning,
uataq.errors.DroppedPollutantWarning(aUserWarning, apart from
EmptyResultWarningso each can be silenced alone).get_recent_obsand
MobileSite.get_obswarn the same way. A pollutant that another instrument
still measured is not left out,get_obs()with no pollutants named does not
warn, and a read where no pollutant has data still raises the error (#63). - Raw
lgr_uggafiles are parsed with pandas' C engine instead of its
python engine, about three times faster: 0.2 s to 0.07 s for a day's file,
and 76 to 26 minutes of CPU for the whole archive (19,554 files at six
sites). Every one of those files gives the same table as before. - A raw
lgr_uggaread (WBB, CSP, FRU, HDP, HPL, ROO) opens only the files
of the days in its range, plus the day on either side as before. It opened
every raw file the site has to read its header, before the range was
applied: a one-day read at WBB, the first on a node, took nearly 5 minutes,
4 of them opening its 4,472 files; it now takes about 30 s, most of it
listing the folders. The same files are read and the same rows returned; a
file outside the range whose header cannot be read is no longer reported. - A raw
lgr_uggaread lists only the day folders that overlap its range (the
range plus the day on either side), where it walked every day folder the site
has (4,336 at WBB) and read the date from every file name. The first read of
a day at WBB in a new process took 2.1 to 3.3 s, 1.1 to 2.2 s of it listing
the folders (22 to 26 s on a node where the file server had not cached them),
and takes 0.4 to 0.5 s, 0.1 s of it listing. The same files ar...
v2026.10.0
The first release since 2025.11.0.
Added
- Research groups per instrument.
groupcan map each instrument to a
group, and with nogroupeach part of a time range is read from the group
that has it (group_datesinconfig.json): TRAX ozone comes from lin before
2017-10-28 and from Horel after. uataq.pollutants: helpers for UATAQ's concentration column names
(CO2d_ppm_caland the like).uataq.gps: great-circle helpers andestimate_speed_course.GPS.read_data
fills missingSpeed_m_sandCourse_degfrom the positions and marks them
inSpeed_EstimatedandCourse_Estimated. Recorded values are never
overwritten.- Horel raw GPS reads carry
GPS_Time_UTC, the receiver's own time. The Horel
logger's clock runs 1 to 20 s ahead of GPS time. Rows without a fix get NaT. - Mobile observations carry
GPS_Group, the group whose GPS located each row. Network.get_availabilityandNetwork.plot_availability: data availability
by site and level.
Changed
- Mobile observations are located only with the GPS of the group they were
read from (breaking): lin rows on the Pi's clock, Horel rows on the logger's
clock. Naming another group's GPS for some rows raisesValueError. Horel rows
used to be joined to lin's GPS, which put moving TRAX rows a median 120 m from
where they were measured and dropped rows. - Time ranges are half-open (breaking): a range's stop is excluded, in row
slicing and inTimeRangemembership. A date string still covers its whole
day. - Network observations keep
_caland_rawcolumns and have an
instrumentcolumn (breaking), so two instruments measuring the same species
no longer produce duplicate columns. - A time range that only touches an instrument's installation or removal date
raisesInactiveInstrumentError. It used to return empty data. - uataq no longer turns on pandas' copy-on-write option. Setting it changed
pandas for the caller's whole session. - Notices are logged, not printed.
- Off CHPC,
import uataqfails after 10 s with a clear message when it cannot
fetch the pipeline's configuration. It used to hang. - Requires Python 3.11 or newer (breaking): Python 3.10 reaches its end of
life in October 2026. Tested on 3.11 through 3.14. - The version now comes from git tags (setuptools-scm). An install from git
between releases reports a development version such as
2025.11.1.dev3+g1a2b3c4rather than the last release's number. - The documentation no longer describes class attributes and methods twice: the
API pages drop a class docstring'sMethodssection and theAttributes
entries that the page documents anyway. Entries that described methods that no
longer exist (Site.read_obs) are gone from the pages.
Fixed
- Horel GPS speed is read as m/s. It was treated as knots, so speeds came out at
about half their value. - lin pipeline files: the header line is skipped. It was read as a row, which
made every column text, and a repeated value in the first row of a headerless
file was altered. - Horel finalized rows with no observations are dropped.
- lin MetOne air-trend files are read with their own column layout. They were
matched to the met sensor's, which lost the PM2.5 column. - Raw LGR files are dated from the style of their file name, not from a table
of analyzer software versions, so an unknown version no longer stops a read. TimeRangekeeps the minutes and seconds of a string and rejects malformed
strings.Site.get_obsmatches pollutant names in any case.- Reads are confined to the time an instrument was installed.
- A read raises
ReaderErrorwhen none of its files can be parsed, and skips a
level directory the instrument does not have. - Raw MetOne pressure columns are named in hPa, as the qaqc and final levels
name them. num_processes="max"sizes its pool by the CPUs the process may use.- Parsing a file no longer fails on Windows when the file and the archive root
are on different drives.
uataq 2025.11.0
First tagged release since the initial v2024.08 snapshot, and the first release archived on Zenodo for a DOI. Highlights since v2024.08: installable/editable packaging, per-group filesystem readers (Horel/Lin groupspaces), Network for aggregating a pollutant across sites, the Horel-group SODAR reader (moved from lair.mesowest), and a growing pytest suite with Codecov coverage reporting.
See the full commit history for details: v2024.08...v2025.11.0