Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update dc2_object_run2.2i_dr6_v2_with_addons #595

Merged
merged 4 commits into from
Jun 2, 2022

Conversation

yymao
Copy link
Member

@yymao yymao commented May 27, 2022

In this PR:

  • The original dc2_object_run2.2i_dr6_v2_with_addons was renamed to dc2_object_run2.2i_dr6_v2_with_addons_v1
  • dc2_object_run2.2i_dr6_v2_with_addons_v2 was created with the new truth match addon
  • dc2_object_run2.2i_dr6_v2_with_addons was changed to an alias config, currently still pointing to dc2_object_run2.2i_dr6_v2_with_addons_v1 so that it doesn't break anyone's one. A deprecation message was added to encourage users to update to dc2_object_run2.2i_dr6_v2_with_addons_v2.

Thanks @johannct for suggestion.

@yymao yymao requested a review from JoanneBogart May 27, 2022 22:09
@yymao
Copy link
Member Author

yymao commented May 27, 2022

In aefba29

  • dc2_run2.2i_truth_merged_summary was added as an alias to dc2_run2.2i_truth_merged_summary_v1-0-0
  • dc2_truth_run2.2i_summary_tract_partition was deprecated

@yymao
Copy link
Member Author

yymao commented May 27, 2022

In 812e6f8,

  • desc_dc2_run2.2i_dr6_v4_object, desc_dc2_run2.2i_dr6_v4_truth, and desc_dc2_run2.2i_dr6_v4_object_with_truth_match were added. Note that v3 is skipped to match the release note version.
  • desc_dc2_run2.2i_dr6_object, desc_dc2_run2.2i_dr6_truth, and desc_dc2_run2.2i_dr6_object_with_truth_match now point to v4 instead of v2.

@JoanneBogart
Copy link
Contributor

I haven't tried to load these, but overall the changes and additions look reasonable. I have a question about the update to dc2_truth_run2.2i_summary_tract_partition.yaml. Base directory is ..truth/tract_partition/raw, which includes rather large parquet files. The suggestion is to use dc2_run2.2i_truth_merged_summary instead. That points to the new, much smaller summary files. Is that because the files in tract_partition/raw include all the columns from galaxy truth? But we don't expect people to actually be referencing those extra columns?

@yymao
Copy link
Member Author

yymao commented May 31, 2022

Checking the size:

  • /global/cfs/cdirs/lsst/shared/DC2-prod/Run2.2i/truth/tract_partition/raw is 73 GB
  • /global/cfs/cdirs/lsst/shared/DC2-prod/Run2.2i/truth/truth_merged_summary_v1-0-0 is 57 GB

Neither of these includes all the columns from galaxy truth. I think the difference mainly come from (1) the smaller footprint of the star truth catalog and (2) the small total number of columns (mostly from removing flux_noMW).

@JoanneBogart
Copy link
Contributor

I was comparing only individual files; I think 2) is the reason for the difference I saw.

Copy link
Contributor

@JoanneBogart JoanneBogart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This all seems fine.

@yymao yymao merged commit 7a3fc84 into master Jun 2, 2022
@yymao yymao deleted the u/yymao/new-truth-merged-catalogs branch June 2, 2022 01:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants