Skip to content

v0.5.0

Latest

Choose a tag to compare

@github-actions github-actions released this 16 Sep 21:30
· 2 commits to main since this release
e829754

panoptes-data 0.5.0

This release moves the package onto the documents panoptes-pipeline writes
and the parquet index built over them, and gives it a query surface that can
express benchmark selection. It is mostly breaking.

Metadata is read from the processed tree -- an observation.json per sequence,
a metadata.json per frame -- and from observations.parquet /
frames.parquet, rather than from a Firestore-derived observations.csv and a
cloud function. Both were downstream of a pipeline that stopped producing them.
There is no fallback: it is not that the cloud sources are deprecated, it is
that nothing makes them.

Set PANOPTES_PROCESSED_ROOT to the document tree, PANOPTES_INDEX_ROOT if
the index sits elsewhere, and PANOPTES_ARCHIVE_ROOT to a local copy of the
raw frames. The three are different trees and conflating them is the easiest
mistake to make here.

  • Every column a caller touches is renamed to the contract's flattened
    document names: uid is image_uid, num_images is num_frames,
    coordinates.mount_ra is mount_ra, time is sequence_time. Joined with
    _ and never ., because a dotted name is a view over a nested map rather
    than storage.

  • search_observations takes keyword arguments only. It has eighteen
    parameters, and inserting one used to rebind every positional argument after
    it.

  • min_num_images is min_num_frames, and the status argument is gone;
    num_usable is the better question.

  • download_images and the download CLI command raise. Archived frames
    404 anonymously in every era of the bucket, and there is no public
    replacement. Frames are read from a local archive instead.

  • SurveySettings.img_metadata_url and observations_url are removed.

  • The two example notebooks are removed; their worked examples are in the
    README.

  • All-sky search. A position is optional, so a unit and a date range, or a
    frame count and a duration, are each a complete query.

  • Benchmark cuts: min_num_usable, min_duration_minutes, field_name,
    camera_id, and a trailing query string. num_usable counts frames the
    pipeline processed cleanly, as distinct from frames that exist.

  • duration in place of end_date: "90 days", "6 months before",
    "10 days before and after", or a timedelta.

  • iso, airmass, moonfrac, moonsep attached to every sequence, since
    observations.parquet has no column for a per-frame reading.

  • find_simultaneous, pairing sequences of one field recorded at the same
    time by different cameras or units, on interval overlap rather than a
    calendar night. Plus a pairs CLI command.

  • duration_minutes and a total_exptime that is populated for the long
    sequences the old summary left null.

  • get_metadata raises on a partial read rather than wrapping the whole loop
    in except Exception: pass, which made half the archive and all of it the
    same return value.

  • The search and get-metadata CLI commands no longer bind one short flag to
    two options: -s set the end date and -r set the radius.

  • A search cone spanning 0h RA finds sequences again, and is widened per
    sequence by that sequence's own pointing drift.

  • A frame document that is missing or unreadable raises rather than yielding a
    quietly shorter observation.

The full list is in CHANGELOG.md.