Make MSNoise a real Python Package

[ThomasLecocq]

MSNoise should be importable like any other python package : installable using package managers, pip, easy_install, eggs or whatever.

This requires:

• Moving all codes in main() functions, calling main() from if name == 'main'.
• Propagating the db.ini path location to all codes so they find the right location
• really using the output_directory (currently only used for CROSS_CORRELATIONS when keep_all is True
• Adding information about that in the documentation
• Provide examples

This will allow MSNoise to be installed by sysadmins, and used by users, with no output placed in the folder of the package.

[ThomasLecocq]

to run MSNoise, my current idea is either to execute steps like this

(1.1) $ python -m msnoise --step 1
or
(1.2) $ python -m msnoise.s01scan_archive

or create a msnoise in Python/Scripts and

(2) $ msnoise --step 1

any suggestions ?

I remember reading some comment advocating for the first solution (written by someone fed up with the /scripts folder being super full)... But with the first (1.1) solution, it's not really easy to pass arguments to step 1...

[krischer]

The first version is nice for commands that are rarely called by users (e.g. tests, benchmarks, ...).

MSNoise on the other hand really focuses on actually using these commands so it is completely justified in my opinion to have a central msnoise script that can (after installation) be called from everywhere.

Then you can design a git like interface (main command followed by the subcommand followed by the arguments), e.g.

$ msnoise scan_archive .
$ msnoise do_other_stuff --who=me /path/to/stuff

That is what I did here: http://lasif.net/cli.html (the documentation is actually autogenerated from the code with the help of a custom sphinx directive). My favorite part turned out to be the testing. A custom pytest fixture (https://github.com/krischer/LASIF/blob/5ad0ff11abe7703c3a9ae4507374389dee4e194f/lasif/tests/testing_helpers.py#L69-106) is used to make it really easy and clean to test the command line interface, e.g. https://github.com/krischer/LASIF/blob/5ad0ff11abe7703c3a9ae4507374389dee4e194f/lasif/tests/test_cli_interface.py#L108-L115

I coded the command dispatching and all the other stuff by hand: It is fairly straightforward to implement some nice features like fuzzy subcommand matching and pretty help messages. But nowadays I would just use something like this: http://click.pocoo.org

[ThomasLecocq]

Mmmm Thanks ! That sound reallllly cool : http://click.pocoo.org/3/quickstart/#nesting-commands

[ThomasLecocq]

Ok, so far so good, using Click

[test1] C:\Users\tlecocq\Desktop>msnoise
Usage: msnoise-script.py [OPTIONS] COMMAND [ARGS]...

Options:
  -t, --threads INTEGER  Nulmber of threads to use
  --help                 Show this message and exit.

Commands:
  bugreport
  config
  install
  scan_archive

[test1] C:\Users\tlecocq\Desktop>msnoise bugreport --help
Usage: msnoise-script.py bugreport [OPTIONS]

Options:
  -s, --sys      System Info
  -m, --modules  Modules Info
  -e, --env      Environment Info
  -a, --all      All Info
  --help         Show this message and exit.

Cooooool :-)

[ThomasLecocq]

[test1] C:\Users\tlecocq\Desktop\tmprun>msnoise
Usage: msnoise-script.py [OPTIONS] COMMAND [ARGS]...

Options:
  -t, --threads INTEGER  Number of threads to use (only affects modules that
                         are designed to do parallel processing)
  --help                 Show this message and exit.

Commands:
  bugreport     This command launches the Bug Report script.
  compute_cc    Computes the CC jobs (based on the "New Jobs"...
  compute_dtt   Computes the dt/t jobs based on the new MWCS...
  compute_mwcs  Computes the MWCS based on the new stacked...
  config        This command launches the Configurator.
  install       This command launches the installer.
  new_jobs      Determines if new CC jobs are to be defined
  plot          Top level command to trigger different plots
  populate      Rapidly scan the archive filenames and find...
  reset         Resets the job to "T"odo.
  scan_archive  Scan the archive and insert into the Data...
  stack         Stacks the [REF] and/or [MOV] windows

almost done :-)

[ThomasLecocq]

f945ba2 marks the first big (read -HUGE-) step toward packaging MSNoise !

[ThomasLecocq]

I think this can be considered done...

[krischer]

I don't think this is really done TBH. For example the dependencies are not specified in the setup.py and I am also not sure it installs everything it needs. A good test is always to install (without --develop or -e) to a clean virtual environment that only contains numpy and see if everything still works and if the tests can somehow be executed.

[ThomasLecocq]

aaah that... is correct :-)

[ThomasLecocq]

and at some point, when that looks ready, learn how to make a PyPi upload... :s

[ThomasLecocq]

I think it's OK...