Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Predict catalog given DECaLS survey image #840

Merged
merged 2 commits into from
Jul 3, 2023
Merged

Predict catalog given DECaLS survey image #840

merged 2 commits into from
Jul 3, 2023

Conversation

zhixiangteoh
Copy link
Contributor

@zhixiangteoh zhixiangteoh commented Jun 26, 2023 •
edited
Loading

  • Uses pyvo.dal SIA service to download image cutouts given RA, Dec, and size in width, height Downloads coadded brick image data
  • Downloads associated tractor catalog from DECaLS server
  • Workflow is (very roughly) DECaLS.prepare_data() > DECaLS.predict_dataloader() > Encoder.predict_step(), similar to SDSS
  • Some fixes for bugs due to encoder assuming always multi-band image data.

Closes #772.

@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

jeff-regier
jeff-regier reviewed Jun 26, 2023
View reviewed changes
bliss/predict.py Outdated Show resolved Hide resolved
jeff-regier
jeff-regier reviewed Jun 26, 2023
View reviewed changes
bliss/surveys/decals.py
return {
"image": image,
"background": np.zeros_like(image), # TODO: find a way to get background
"wcs": wcs,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we read in the PSF somewhere too? I think we'd need the DECaLS PSFs to generate realistic training data.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DECaLS data generation - punting to after #853.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tracked by #871

jeff-regier
jeff-regier reviewed Jun 26, 2023
View reviewed changes
bliss/conf/base_config.yaml
# height: 1489
# bands: ["g", "r", "z"]
# predict_device: ${predict.device}
# predict_crop: ${predict.crop}
trainer: ${training.trainer}
encoder: ${encoder}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'd also likely need a simulator to generate data that resembles decals data, i.e. which samples backgrounds and PSFs from DECaLs data. (The simulator above has an sdss_fields entry -- but ideally the simulator wouldn't be specific to any one survey.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DECaLS data generation - punting to after #853.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tracked by #871

jeff-regier
jeff-regier requested changes Jun 26, 2023
View reviewed changes
Copy link
Contributor

@jeff-regier jeff-regier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! Our codebase will be so extensible/maintainable once we abstract away the survey-specific code, confining it to the surveys module.

@zhixiangteoh
Predict catalog given DECaLS survey image
63bb52e 
- Uses pyvo.dal SIA service to download image cutouts given RA, Dec, and size in
  width, height
- Determines corresponding brick name, and downloads associated tractor catalog
  from DECaLS server
- Workflow is (very roughly) DECaLS.prepare_data() > DECaLS.predict_dataloader()
  > Encoder.predict_step(), similar to SDSS

Closes #772.
@zhixiangteoh zhixiangteoh marked this pull request as ready for review July 3, 2023 05:13
@zhixiangteoh
Copy link
Contributor Author

@jeff-regier ptal in general
@sawanp813 ptal at encoder changes to allow single-band inference (specifically, changes to encoder.py and the new test test_predict.py's TestPredict::test_predict_sdss_single_band)
@aakashdp6548 ptal at changes to mocktests.py and how MockSDSSDownloader is being used

@codecov
Copy link

codecov bot commented Jul 3, 2023 •
edited
Loading

Codecov Report

Merging #840 (75b9240) into master (d639fcd) will decrease coverage by 0.85%.
The diff coverage is 85.18%.

@@            Coverage Diff             @@
##           master     #840      +/-   ##
==========================================
- Coverage   96.34%   95.50%   -0.85%     
==========================================
  Files          19       19              
  Lines        2024     2157     +133     
==========================================
+ Hits         1950     2060     +110     
- Misses         74       97      +23
Flag Coverage Δ
unittests 95.50% <85.18%> (-0.85%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
bliss/utils/download_utils.py 80.00% <46.15%> (-20.00%) ⬇️
bliss/surveys/decals.py 92.26% <87.12%> (-6.31%) ⬇️
bliss/predict.py 97.59% <88.23%> (-2.41%) ⬇️
bliss/encoder.py 94.84% <100.00%> (+0.05%) ⬆️
bliss/surveys/sdss.py 95.86% <100.00%> (-0.09%) ⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@zhixiangteoh
Do all DECaLS tasks in brick units
75b9240 
Notes:
- Brick images are coadded images
- Tractor catalog is based on single-epoch images (cf. note in
  "https://www.legacysurvey.org/dr10/files/#image-stacks-south-coadd")
- Fix bugs from encoder assuming multi-band images
aakashdp6548
aakashdp6548 approved these changes Jul 3, 2023
View reviewed changes
Copy link
Collaborator

@aakashdp6548 aakashdp6548 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test stuff looks good to me. DECaLS stuff looks good overall too but I didn't review it too in-depth

jeff-regier
jeff-regier approved these changes Jul 3, 2023
View reviewed changes
Copy link
Contributor

@jeff-regier jeff-regier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. And I'm looking forward to the PR resolving #853 !

@jeff-regier jeff-regier merged commit 4f41ffd into master Jul 3, 2023
@jeff-regier jeff-regier deleted the 772-infer-decals branch July 3, 2023 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aakashdp6548 aakashdp6548 aakashdp6548 approved these changes

@jeff-regier jeff-regier jeff-regier approved these changes

@sawanp813 sawanp813 Awaiting requested review from sawanp813

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

infer catalogs based on DECaLS images
3 participants
@zhixiangteoh @jeff-regier @aakashdp6548