Repository navigation
panoptes-data 0.5.0
This release moves the package onto the documents panoptes-pipeline writes
and the parquet index built over them, and gives it a query surface that can
express benchmark selection. It is mostly breaking.
Metadata is read from the processed tree -- an observation.json per sequence,
a metadata.json per frame -- and from observations.parquet /
frames.parquet, rather than from a Firestore-derived observations.csv and a
cloud function. Both were downstream of a pipeline that stopped producing them.
There is no fallback: it is not that the cloud sources are deprecated, it is
that nothing makes them.
Set PANOPTES_PROCESSED_ROOT to the document tree, PANOPTES_INDEX_ROOT if
the index sits elsewhere, and PANOPTES_ARCHIVE_ROOT to a local copy of the
raw frames. The three are different trees and conflating them is the easiest
mistake to make here.
-
Every column a caller touches is renamed to the contract's flattened
document names:uidisimage_uid,num_imagesisnum_frames,
coordinates.mount_raismount_ra,timeissequence_time. Joined with
_and never., because a dotted name is a view over a nested map rather
than storage. -
search_observationstakes keyword arguments only. It has eighteen
parameters, and inserting one used to rebind every positional argument after
it. -
min_num_imagesismin_num_frames, and thestatusargument is gone;
num_usableis the better question. -
download_imagesand thedownloadCLI command raise. Archived frames
404 anonymously in every era of the bucket, and there is no public
replacement. Frames are read from a local archive instead. -
SurveySettings.img_metadata_urlandobservations_urlare removed. -
The two example notebooks are removed; their worked examples are in the
README. -
All-sky search. A position is optional, so a unit and a date range, or a
frame count and a duration, are each a complete query. -
Benchmark cuts:
min_num_usable,min_duration_minutes,field_name,
camera_id, and a trailingquerystring.num_usablecounts frames the
pipeline processed cleanly, as distinct from frames that exist. -
durationin place ofend_date:"90 days","6 months before",
"10 days before and after", or atimedelta. -
iso,airmass,moonfrac,moonsepattached to every sequence, since
observations.parquethas no column for a per-frame reading. -
find_simultaneous, pairing sequences of one field recorded at the same
time by different cameras or units, on interval overlap rather than a
calendar night. Plus apairsCLI command. -
duration_minutesand atotal_exptimethat is populated for the long
sequences the old summary left null. -
get_metadataraises on a partial read rather than wrapping the whole loop
inexcept Exception: pass, which made half the archive and all of it the
same return value. -
The
searchandget-metadataCLI commands no longer bind one short flag to
two options:-sset the end date and-rset the radius. -
A search cone spanning 0h RA finds sequences again, and is widened per
sequence by that sequence's own pointing drift. -
A frame document that is missing or unreadable raises rather than yielding a
quietly shorter observation.
The full list is in CHANGELOG.md.