Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ENH: Logging for skypy command line script and Pipeline class #453

Merged
merged 16 commits into from
Apr 22, 2021
Merged

ENH: Logging for skypy command line script and Pipeline class #453

merged 16 commits into from
Apr 22, 2021

Conversation

rrjbca
Copy link
Contributor

@rrjbca rrjbca commented Mar 17, 2021 •
edited
Loading

Description

  • Setup logger, stream handler and formatter for skypy command line script
  • Set the logging level using a command line argument
  • Log any exceptions raised while executing the command line script
  • Setup a logger for the skypy.pipeline module
  • Log each job and function call in the Pipeline class and when writing the output file

Example log output:

>>> skypy -v examples/galaxies/sdss_photometry.yml galaxies.fits
2021-03-22 19:48:35,758 [INFO] skypy.pipeline: Setting cosmology
2021-03-22 19:48:35,758 [INFO] skypy.pipeline: Calling get
2021-03-22 19:48:35,758 [INFO] skypy.pipeline: Generating red_galaxies
2021-03-22 19:48:35,758 [INFO] skypy.pipeline: Calling Table
2021-03-22 19:48:35,758 [INFO] skypy.pipeline: Generating blue_galaxies
2021-03-22 19:48:35,758 [INFO] skypy.pipeline: Calling Table
2021-03-22 19:48:35,759 [INFO] skypy.pipeline: Generating sky_area
2021-03-22 19:48:35,759 [INFO] skypy.pipeline: Generating magnitude_limit
2021-03-22 19:48:35,759 [INFO] skypy.pipeline: Generating phi_star_red
2021-03-22 19:48:35,759 [INFO] skypy.pipeline: Calling Exponential1D
2021-03-22 19:48:35,759 [INFO] skypy.pipeline: Generating M_star_red
2021-03-22 19:48:35,759 [INFO] skypy.pipeline: Calling Linear1D
2021-03-22 19:48:35,760 [INFO] skypy.pipeline: Generating phi_star
2021-03-22 19:48:35,760 [INFO] skypy.pipeline: Calling Exponential1D
2021-03-22 19:48:35,760 [INFO] skypy.pipeline: Generating M_star
2021-03-22 19:48:35,760 [INFO] skypy.pipeline: Calling Linear1D
2021-03-22 19:48:35,761 [INFO] skypy.pipeline: Generating z_range
2021-03-22 19:48:35,761 [INFO] skypy.pipeline: Calling linspace
2021-03-22 19:48:35,761 [INFO] skypy.pipeline: Generating red_galaxies.redshift, magnitude
2021-03-22 19:48:35,761 [INFO] skypy.pipeline: Calling schechter_lf
2021-03-22 19:48:36,776 [INFO] skypy.pipeline: Generating red_galaxies.spectral_coefficients
2021-03-22 19:48:36,776 [INFO] skypy.pipeline: Calling dirichlet_coefficients
2021-03-22 19:48:36,824 [INFO] skypy.pipeline: Generating red_galaxies.stellar_mass
2021-03-22 19:48:36,824 [INFO] skypy.pipeline: Calling stellar_mass
2021-03-22 19:48:36,848 [INFO] skypy.pipeline: Generating red_galaxies.mag_u, mag_g, mag_r, mag_i, mag_z
2021-03-22 19:48:36,848 [INFO] skypy.pipeline: Calling apparent_magnitudes
2021-03-22 19:48:45,088 [INFO] skypy.pipeline: Generating blue_galaxies.redshift, magnitude
2021-03-22 19:48:45,088 [INFO] skypy.pipeline: Calling schechter_lf
2021-03-22 19:48:47,656 [INFO] skypy.pipeline: Generating blue_galaxies.spectral_coefficients
2021-03-22 19:48:47,656 [INFO] skypy.pipeline: Calling dirichlet_coefficients
2021-03-22 19:48:47,781 [INFO] skypy.pipeline: Generating blue_galaxies.stellar_mass
2021-03-22 19:48:47,781 [INFO] skypy.pipeline: Calling stellar_mass
2021-03-22 19:48:47,793 [INFO] skypy.pipeline: Generating blue_galaxies.mag_u, mag_g, mag_r, mag_i, mag_z
2021-03-22 19:48:47,793 [INFO] skypy.pipeline: Calling apparent_magnitudes
2021-03-22 19:48:57,491 [INFO] skypy: Writing galaxies.fits

Checklist

  • Follow the Contributor Guidelines
  • Write unit tests
  • Write documentation strings
  • Assign someone from your working team to review this pull request
  • Assign someone from the infrastructure team to review this pull request

@rrjbca rrjbca added enhancement Improvement of existing feature infrastructure module: pipeline labels Mar 17, 2021
@rrjbca rrjbca marked this pull request as ready for review March 18, 2021 10:36
@rrjbca rrjbca requested a review from a team as a code owner March 18, 2021 10:36
ntessore
ntessore reviewed Mar 18, 2021
View reviewed changes
Copy link
Member

@ntessore ntessore left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I generally prefer repeatable -v,--verbose and -q,--quiet options but that is a matter of personal taste.

skypy/pipeline/_pipeline.py Outdated Show resolved Hide resolved
skypy/pipeline/scripts/skypy.py Outdated Show resolved Hide resolved
rrjbca
rrjbca commented Mar 29, 2021
View reviewed changes
docs/pipeline/index.rst Outdated Show resolved Hide resolved
@rrjbca rrjbca merged commit 1281106 into skypyproject:main Apr 22, 2021
@rrjbca rrjbca deleted the logging branch April 22, 2021 14:11
@rrjbca rrjbca mentioned this pull request Jun 30, 2021
Closed
40 tasks
itrharrison added a commit that referenced this pull request Dec 15, 2022
@itrharrison @rrjbca
@ntessore @Lucia-Fonseca @sutieng @philipp128 @Fox-Davidson
MAINT: merge main into module/halos (#581)
3496ac0 
* Update name of default branch to main (#434)

* update mailmap (#432)

* Write all tables to a single FITS/HDF5 file (#425)

* ADR 3: Position sampling and patches of the sky (#422)

* BUG: Raise ImportError if optional dependency speclite is not installed (#437)

* MAINT: Set NumPy latest supported version to 1.20 #440

* Update status badges (#441)

* MAINT: Update Lucia affiliation (#451)

* MAINT: add SIT's information (#450)

* DOC: Fix contributor guidelines link (#449)

Co-authored-by: Richard R <58728519+rrjbca@users.noreply.github.com>
Co-authored-by: Nicolas Tessore <n.tessore@ucl.ac.uk>

* ENH: Logging for skypy command line script and Pipeline class (#453)

* DOC: Describe speclite filters in documentation (#457)

* ENH: Config syntax for importing objects (#463)

* DOC: List of Features (#456)

* DOC: How to construct config files (#454)

* DOC: Remove docstring examples (#429)

* MAINT: Update Zenodo entry for RPR (#468)

* DOC: Readme updates (#460)

* DOC: Expanded landing page documentation (#228)

* DOC: Inverse transform sampling accuracy warning (#472)

* MAINT: Set astropy latest supported version to 4.2 (#483)

* DOC: zenodo json members update (#481)

* DOC: Ryden04 ellipticity doc missing section (#477)

* MAINT: Update numpy and scipy latest supported versions (#488)

* BUG: Change invalid ecsv datatype from int to uint16 (#485)

* DEV: setuptools==58.0.0 (#493)

Co-authored-by: Nicolas Tessore <n.tessore@ucl.ac.uk>

* Add compatibility workflow badge (#487)

* DEV: Enable pip to install pre-releases in the tox dev environments (#491)

* TST: Use tmp_path fixture for temporary files in unit tests (#489)

* BUG: Move handling of context arguments after handling of .depends keyword (#465)

* BLD: Set astropy latest supported version to 4.3 and speclite minversion to 0.14 (#486)

* REV: restore setuptools to latest version on readthedocs (#494)

* DEV: pyparsing<3.0.0 (#500)

* Check new astropy file overwrite error message in logging test (#498)

* REV: restore pyparsing to latest version for doc builds (#501)

* DOC: Update citation file with JOSS paper reference (#496)

* BLD: Set astropy latest supported version to 5.0 (#504)

* BLD: Set python latest supported version to 3.10 (#505)

* BLD: Set numpy latest supported version to 1.22 (#506)

* BLD: Set python oldest supported version to 3.7 (#507)

* DOC: Fix code of conduct link (#508)

* Changed y-label in luminosity function example. (#512)

* BLD: Set scipy latest supported version to 1.8 (#510)

* ENH: Rykoff model of the magnitude uncertainty (#526)

* TST: assert photometric error is numerically close to the analytic value (#545)

* TST: Drop deprecated astropy.tests.helper.raises (#546)

* ENH: compute kcorrect remaining stellar mass (#476)

* compute kcorrect remaining stellar mass

* added test for stellar mass remain

Co-authored-by: Ian Harrison <itrharrison@gmail.com>
Co-authored-by: Lucia F. de la Bella <55983939+Lucia-Fonseca@users.noreply.github.com>

* ENH: Logistic completeness function (#521)

* BLD: Set astropy latest supported version to 5.1 (#547)

* BUG: `schechter_smf` callable input and docs (#525)

* DOC: Typo in Rykoff error (#550)

* add Fox's details (#551)

Co-authored-by: Richard R <58728519+rrjbca@users.noreply.github.com>

* BLD: Set numpy latest supported version to 1.23 (#552)

* codestyle fixes

* add six requirement for colossus

* tried to fix docs builds

* update passenv

* rtd configuration

Co-authored-by: Richard R <58728519+rrjbca@users.noreply.github.com>
Co-authored-by: Nicolas Tessore <n.tessore@ucl.ac.uk>
Co-authored-by: Lucia F. de la Bella <55983939+Lucia-Fonseca@users.noreply.github.com>
Co-authored-by: Sut-Ieng Tam <30295725+sutieng@users.noreply.github.com>
Co-authored-by: philipp128 <48715661+philipp128@users.noreply.github.com>
Co-authored-by: Fox Davidson <93545862+Fox-Davidson@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ntessore ntessore ntessore approved these changes

Assignees
No one assigned
Labels
enhancement Improvement of existing feature infrastructure module: pipeline
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@rrjbca @ntessore