Skip to content

Refactors to support DP1 import#55

Merged
rknop merged 37 commits intomainfrom
u/rknop/dp1-parquet
Aug 15, 2025
Merged

Refactors to support DP1 import#55
rknop merged 37 commits intomainfrom
u/rknop/dp1-parquet

Conversation

@rknop
Copy link
Copy Markdown
Contributor

@rknop rknop commented Jul 23, 2025

Massive refactors:

  • diasoruceid and diaforcedsourceid have gone away. They weren't unique in DP1, and we learned that they were supposed to have gone away anyway. Those tables now use (diaobjectid,visit,processing_version) as their compound primary key. (We are told that objectid will be globally unique. Some of the bits of objectid encode the data release, so the same number won't show up in more than one data release. However, the same actual on-sky object will still have more than one objectid, as it will have different values in different data releases!)
  • the visit column of various tables had to be convered from integer → bigint (since they are bigints in DP1!)
  • Changed the avro schema (now namespace version 0.2, used to be 0.1) to reflect the removal of the dia*sourceid columns
  • Refactor of SNANA loader to handle changed schema, but also to share some code with the all-new DP1 parquet file loader
  • Removed the "snapshot" database tables. This isn't really part of this PR, but removing them made more sense than editing the code to handle them, since we're going to need to get rid of them anyway. (Cf: discussion during fastdb week with Bruno re: how scary those tables were going to get.) This will be dealt with in the future: see Issue Processing Version refactor #52.
  • Lots of library, web server and test code had to be updated to handle the schema changes. OMG getting all the tests to pass....

@rknop rknop changed the title Refactors to support DP1 inport Refactors to support DP1 import Jul 23, 2025
@rknop rknop mentioned this pull request Jul 22, 2025
@rknop rknop merged commit b11bbdf into main Aug 15, 2025
2 checks passed
@rknop rknop deleted the u/rknop/dp1-parquet branch August 27, 2025 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant