Modules to clean up

[aguinot]

Here is a list of what I think are the easiest and first module to clean up :

• SExtractor_runner / SExtractor_runner_exp (Cleanup sextractor #412)
• PSFEx_runner (Cleanup psfex #413)
• vignetmaker_runner / vignetmaker_runner2 (vignetmaker Clean Up #411)
• split_exp_runner (split exposure runner standardisation #410)
• core pipeline (Multiple Module Runs #415)
• make_catalog_runner (Make Catalog Clean Up #435)
• SETools (SETools Clean Up #430)
• psfex_interp_runner (PSFEx Interpolation Clean Up #434)
• MCCD (MCCD cleanup + MCCD plot package: Adding rejection of objects with zero size that are not flagged #431)
• find_exposures_runner (Find exposures revision #423)
• get_images_runner (Get images Runner revision #429)
• uncompress_fits_image_runner (uncompress_fits_image module revision #427 )
• spread_model_runner (spread_model revision, suggestion of common module format #403)

[sfarrens]

Module Clean Up Guidelines

File Structure

For each module there should be:

• A file called <MODULE NAME>_runner.py in shapepipe/modules
• A directory called <MODULE NAME>_package in shapepipe/modules

Module Runner

The module runner file should contain:

• A clear docstring explaining what the module does
• The import of the module runner decorator

from shapepipe.modules.module_decorator import module_runner

• The import of the module package interface

from shapepipe.modules.<MODULE NAME>_package.<PYTHON FILE NAME> import <MODULE INTERFACE>

• The import of the execute command (if relevant)

from shapepipe.pipeline.execute import execute

• A function called <MODULE NAME>_runner decorated with the module runner decorator (see examples)

⚠️ Note that the function arguments are always the same

The module runner function should contain:

• Any relevant calls to the configuration file
• Any relevant calls to the module package interface
• Any relevant calls to execute
• A single return statement with either the stdout and stderr returned by the module or simply None, None

The module runner function should NOT contain:

• ❌ Any calculations
• ❌ Any manipulation of data

Module Package

The module package directory should contain:

• An __init__.py file with a descriptive docstring
• A single or multiple python modules required to perform the given task

(One of) the python module(s) should contain:

• A single interface function of class which takes the inputs provided by the pipeline and provides the appropriate module outputs
• In the case of command line executable modules this should simply provide the desired command line

[sfarrens]

@aguinot , @martinkilbinger and @andrevitorelli here are the guidelines for the module clean up. Feel free to post comments if I missed anything important.

You should have already been notified regarding an issue I opened for you.

[martinkilbinger]

Example module, that is already on the master branch: paste_cat_runner

[sfarrens]

Before finalising this activity and merging the open PRs (that have passed the CI tests), I propose we converge on a few style guidelines for ShapePipe. What matters here is not necessarily the way we do things, rather that we are all doing things in the same way. This way we can continue to work on this clean up activity in a consistent way.

Here is my proposal, feel free to comment if you have a counter proposal for any of these points.

Long Lines (>79 characters)

Use () blocks. e.g.

my_short_line = ...
my_long_line = (
   ...
)
my_short_func(arg1, arg2, ...)
my_long_func(
   first_arg=arg1,
   second_arg=arg2,
)

String Concatenation

Switch from .format() to f-string syntax and explicitly concatenate strings with + operator. e.g.

my_short_str = f'This is a short string with param value {param_val}.'
my_long_str = (
   f'This is my long string with several parameters values {param_val1}, '
   + f'{param_val2}, {param_val3}, etc.'
)

Module Runner Decorator

Always explicitly include the module version in the decorator. Other options can be omitted if they use the default values. All modules should be at least 0.1 after clean up. e.g.

@module_runner(
   version=0.1,
   ...
)

[martinkilbinger]

@sfarrens, the suggested style is fine for me.

I was not too familiar with the brackets () to split long lines, I've more seen either backslash \ or no special character at the end of a split line. Are brackets considered more readable ?

Could you please add an example of a long logical expression ? I never know whether to put operators such as ||, &, and, at the end or beginning of a split line.

[sfarrens]

@martinkilbinger

I was not too familiar with the brackets () to split long lines, I've more seen either backslash \ or no special character at the end of a split line. Are brackets considered more readable ?

It appears to be favoured by several Python style guides. I have been sticking to this lately, but I think it is very subjective whether or not it is more or less readable. I think it's probably favoured because it's easier to be consistent.

Could you please add an example of a long logical expression ? I never know whether to put operators such as ||, &, and, at the end or beginning of a split line.

OK, so in general you should be careful using bitwise logical operators like | and & and whenever possible use or and and. Anyway, you can break down a long expression like this e.g.

if (
    (param > 5 and param < 7)
    or (param < 0)
):