Release notes

Release 1.0

Release 1.0.2

Release made on 2013/08/21. This corresponds to the version 760 of the subversion repository. Bug fix for the 1.0.1, the update feature was broken for jobs with Inf (or -Inf) values in their options.

Release 1.0.1

Release made on 2013/08/20. This corresponds to the version 755 of the subversion repository.

• Support of a new field DEP in a job description. This field can be used to explicitly list a list of dependencies (cell of strings with job names). The field DEP is dealt with appropriately by PSOM_MERGE_PIPELINE.
• In qsub/msub modes, job names are now truncated to 8 characters before submission to qsub/msub. This was implemented because long job names crash some flavors of qsub/msub.
• PSOM_CMP_VAR now work on structures with similar fields names but where the field names have not been built in the same order (i.e. the function is insensitive to the order of field names). This impacts only octave, as in matlab fields are ordered alphabetically, independently of the order of creation.
• PSOM_CMP_VAR now has options to control the level of tolerable numerical noise as well as the ability to compare only the fields found in one of the variables (when comparing structures).
• In qsub/msub mode, the subprocesses initiated by PSOM to quickly submit jobs are now terminated cleanly (before it left some zombie jobs behind).
• Minor bug fixes and doc improvement.

Release 1.0(.0)

This corresponds to the version 727 of the subversion repository.

• A new logo for PSOM. I had sweared never to become a computer scientist that kind-of want to be a designer. I lost.
• Added a text file PSOM_news_feeds.csv in the logs folder. This can be easily parsed by third-party software (such as a GUI) to monitor the progress of the execution of a pipeline.
• Restarting a large pipeline is now much faster
• New profiling capabilities: it is now possible to get stats on the execution time of jobs / number of jobs running in parallel etc using PSOM_PIPELINE_VISU
• Failures of job submission are now treated as failures of jobs (in qsub/msub mode)
• Improved verbose (more compact)
• A new function "psom_quarantine_svn" to export all the subversion repositories detected in the search path into designated "quarantine" folder
• Some documentation updates / addition
• Minor bug fixes and improvements

Release 0.9

This corresponds to the version 571 of the subversion repository.

• The [http://code.google.com/p/psom/wiki/ConfigurationPsom PSOM configuration tutorial] has been updated to reflect the changes made in release 0.9.
• The [http://code.google.com/p/psom/wiki/HowToUsePsom PSOM tutorial] has been updated to match the reflect made to psom_demo_pipeline in release 0.9.
• A new wiki page providing [http://code.google.com/p/psom/wiki/CodingGuidelines coding guidelines] for PSOM users.
• PSOM_RUN_JOB can now be invoked to run a job directly without calling PSOM_RUN_PIPELINE.
• A new tool (PSOM_WRITE_PIPELINE2XML) to save a pipeline in a series of .xml+.mat files describing the individual jobs. This was used to interface PSOM with [http://cbrain.mcgill.ca/ CBRAIN].
• A new function (PSOM_SET_RAND_SEED) to set the state of the random number generator, which works in both Matlab and Octave. PSOM now by default sets the seed indepedently for each job based on the clock.
• The function PSOM_FILE_TMP and PSOM_PATH_TMP can be used to generate unique temporary names for files or folders. They are using random strings, and also use the global variable GB_PSOM_JOB_NAME to add the job name in the temporary file/folder names.
• It is now possible to add a prefix to all the jobs of one of the pipeline when merging two pipelines with PSOM_MERGE_PIPELINE.
• A new execution mode : 'background'. In this mode jobs run in the background, and they will get interrupted if the initial matlab/octave session is interrupted (which is not the case in 'batch' mode). The default mode is now 'session' for the pipeline manager, and 'background' for the jobs.
• A new function PSOM_CONFIG. It will test step-by-step that the submission of jobs is successful, and provide context-dependent error messages and suggestions if an issue is detected. This function should help a lot setting up the configuration of PSOM.
• The PSOM has now been tested on a variety of operating systems (Linux, Windows, Mac OSX), with both Matlab and Octave and with different configurations. So far, the only issue that was found was the 'background'/'batch' modes for Octave on Cygwin (Windows).
• Many minor bug fixes and improvements.

Release 0.8

Release 0.8.8

Release made on 2011/01/17. This corresponds to the version 491 of the subversion repository.

• There is a new function PSOM_STRUCT_DEFAULTS which can be used to set the default values of fields in a structure. This method is much faster than PSOM_SET_DEFAULTS which upload fields as variables in memory. This last practice is anyway discouraged by the Mathworks. This new function will thus become the standard method to set default arguments in the development framework.
• A bunch of small modifs that speed up the pipeline initialization and manager big time. Even more improvement is to be expected soon, but PSOM_RUN_PIPELINE is already much smoother now.

Release 0.8.7

This corresponds to the version 477 of the subversion repository.

• There are now commented lines at the end of PSOM_GB_VARS to change the default behaviour of PSOM. These could be added to .octaverc
• The default of OPT.FLAG_PAUSE is now false. That means that the pipeline manager will not wait for confirmation before starting the pipeline.
• Important bug fixes in PSOM_FILE_TMP and PSOM_PATH_TMP, which basically didn't work before. These are not (yet) documented features.
• A new mechanism to describe jobs : FILES_CLEAN. This can be used to list files that will be deleted by a job. This is taken into account to define dependencies (it job A has a file as an input, and job B deletes this file, job B depends on job A).
• A new command PSOM_CLEAN, to delete files. The format of input files is flexible (string, cell of strings, multi-level structure with strings or cell of strings as terminal nodes).
• A new command PSOM_ADD_CLEAN. This adds a job to a pipeline using PSOM_CLEAN.
• PSOM_VISU_DEPENDENCIES now works in Octave. It depends on the package GRAPHVIZ http://www.graphviz.org/

Release 0.8.6

This corresponds to the version 464 of the subversion repository.

• PSOM now creates a global variable names GB_PSOM_NAME_JOB that can be used by the job. In particular, PSOM_FILE_TMP and PSOM_FOLDER_TMP uses this name, which can be handy to generate temporary file names.
• New function PSOM_ADD_JOB that automatizes the task of adding a job to a pipeline (assuming a "brick" syntax).
• It is now possible to create a file psom_gb_vars_local.m anywhere in matlab/octave search path, and this file will override the default configurations found in psom_gb_vars.m
• Bug fix : the matlab initialization sequence was not passed to the jobs. Only the default value was used (so changing psom_gb_vars.m worked ok).
• Code simplification. A bit.
• The pipeline manager now uses the same search path as the jobs (before it was using the one of the calling environment).
• The search path can now be configured with '' to use the current search path, 'gb_psom_omitted' to make no attempt to change the search path (the start-up search path will apply), or any string to explicitly set up the search path.

Release 0.8.5

This corresponds to the version 431 of the subversion repository.

• Folders can now be used in place of file names in Octave (this already worked in matlab).
• The pipeline manager now checks that all jobs have a "command" field. This may seem obvious, but jobs without a command resulted in a weird non-explicit crash of the pipeline.
• Improved verbose when the pipeline ends.
• Deleting the "PIPE.lock" file in the logs folder will now force the pipeline manager to stop, even if it is running in the background. If a bunch of jobs are being submitted, the manager will quit upon completion of submissions.
• The random number generator is now initialized independently for each job using the CPU time. This change impacts Matlab only, as it is the default behaviour in Octave.
• There was a bug in the pipeline update procedure : jobs that had a parameter equal to NaN were always considered as needing to be reprocessed. This was because (NaN==NaN) results into "false" in matlab, against all odds. This should be fixed now (>> psom_cmp_var(NaN,NaN) results in "true").

Release 0.8.4

• Support of PSOM in Matlab for windows, in session and batch modes. This is an unstable version, any feedback will be greatly appreciated.

Release 0.8.3

• A new flag opt.flag_verbose allows to get rid of all the verbose.
• A new flag opt.flag_pause allows to get rid of the pauses in the initialization stage.
• PSOM has now been tested with Octave 3.0.1 and 3.0.4
• A new execution mode : msub. The options of msub are passed through opt.qsub_options.
• Verbose of all pipeline updates when restarting the pipeline manager. The pipeline execution now makes a pause and asks for the user's approval before anything is actually processed.
• A 'time' option in psom_pipeline_visu to display the computation time of all jobs.
• A new function psom_write_dependencies to write dependency graphs in pdf format using the opensource graphviz package.
• Several minor bug fix.

Release 0.8.1/2

• Minor bug fixes and improvements.

Release 0.8

• Differences with version 0.7 are minor.
• The demo psom_demo_pipeline is now complete and the [http://code.google.com/p/psom/wiki/HowToUsePsom PSOM tutorial] is available.

Release 0.7

The update feature is now working, which means that psom_run_pipeline can be executed multiple times in the same log folder, and should deal in a sensible way with changes made to the pipeline. There is also a new logo for the PSOM project.

Release 0.6

Release 0.6.0.3

The way the pipeline manager deals with job status has been completely revised. The pipeline manager is now able to deal easily with pipelines of up to 5000 jobs and probably more.

Release 0.6.0.2

A first successful test on a big pipeline (>5000 jobs with a complex dependency graph) has been done. Learning from that test, a number of improvements have been made to speed up the pipeline manager.

Release 0.6.0.1

• A bunch of bug fixes.
• Release 0.6.0.1 has been tested on Linux and should work.

Release 0.6

• A 'psom_run_pipeline' function, which combines initialization, running and visualization of the pipeline and adapts to the current state of the pipeline.`
• The tests have been limited so far to Matlab and Linux.