preparing for next version release

README.rst

@@ -42,8 +42,8 @@ system at NERSC_.  It is currently being developed on GitHub_.
 Requirements
 ------------
 
-HPSSPy assumes that the HPSS utilities `hsi and htar`_ are installed.  You may
-need a NERSC account to download and install these utilities.
+HPSSPy assumes that the HPSS utilities `hsi and htar`_ are installed.  As of
+2023, these utilities are only available within the NERSC_ environment.
 
 .. _`hsi and htar`: https://docs.nersc.gov/filesystems/archive/#common-commands
 

doc/changes.rst

@@ -2,10 +2,16 @@
 Release Notes
 =============
 
-0.6.2 (unreleased)
+0.7.0 (2023-07-14)
 ------------------
 
-* No changes yet.
+* :command:`missing_from_hpss` will proceed through all stages, even if
+  serious errors are detected, to facilitate batch processing. The older
+  behavior can be enabled with ``--exit-on-error`` (PR `#15`_).
+* Fix error handling for a variety of corner cases (PR `#15`_).
+* Increase test coverage to 100% (PR `#15`_).
+
+.. _`#15`: https://github.com/weaverba137/hpsspy/pull/15
 
 0.6.1 (2022-05-20)
 ------------------

doc/index.rst

@@ -21,10 +21,10 @@ system at NERSC_.  It is currently being developed on GitHub_.
 Requirements
 ++++++++++++
 
-HPSSPy assumes that the HPSS utilities `hsi and htar`_ are installed.  You may
-need a NERSC account to download and install these utilities.
+HPSSPy assumes that the HPSS utilities `hsi and htar`_ are installed.  As of
+2023, these utilities are only available within the NERSC_ environment.
 
-.. _`hsi and htar`: https://www.nersc.gov/users/data-and-file-systems/hpss/storing-and-retrieving-data/software-downloads/
+.. _`hsi and htar`: https://docs.nersc.gov/filesystems/archive/#common-commands
 
 HPSSPy expects these utilities to exist in the directory ``${HPSS_DIR}/bin``, so
 be sure the environment variable :envvar:`HPSS_DIR` is defined.

doc/using.rst

@@ -18,7 +18,7 @@ is equivalent to::
 Options
 +++++++
 
-There are a lot of command-line options.  ``missing_from_hpss --help`` will
+There are several of command-line options.  ``missing_from_hpss --help`` will
 display all of them. Just the short versions of the commands are
 shown here.
 
@@ -27,6 +27,8 @@ shown here.
             allows the user to choose any directory.
 -D          Delete and recreate the disk cache file
             (described below).
+-E          Exit if an error is detected while processing files
+            on disk or on HPSS.
 -H          Delete and recreate the HPSS cache file
             (described below).
 -l N        Limit archive files to this size in GB.
@@ -60,15 +62,15 @@ by default.  The files are:
 Disk Cache
     A CSV file of the form ``disk_cache_<section>.csv``, where ``<section>`` is
     the section (as defined above) specified on the command-line.  The
-    columns are file name and file size in bytes.
+    columns are file name, file size in bytes and modification time.
 
 HPSS Cache
-    A plain-text file of the form ``hpss_cache_<section>.txt``,
-    where ``<section>`` is the section (as defined above) specified on
-    the command-line.  This is simply a list of files found on HPSS.
+    A CSV file of the form ``hpss_cache_<section>.csv``, where ``<section>`` is
+    the section (as defined above) specified on the command-line.  The
+    columns are file name, file size in bytes and modification time.
 
 Missing File Cache
-    A JSON file of the form ``$HOME/cache/missing_files_<section>.json``,
+    A JSON file of the form ``missing_files_<section>.json``,
     where ``<section>`` is the section (as defined above) specified on the
     command-line. It contains a map of HPSS archive files to the files that
     belong in that archive.  In addition the size of the resulting files

hpsspy/__init__.py

@@ -6,7 +6,7 @@
 
 Python interface to the HPSS system.
 """
-__version__ = '0.6.2.dev324'
+__version__ = '0.7.0'
 
 
 class HpssError(Exception):

hpsspy/scan.py

@@ -634,6 +634,9 @@ def _options():
     parser.add_argument('-D', '--overwrite-disk', action='store_true',
                         dest='overwrite_disk',
                         help='Ignore any existing disk cache files.')
+    parser.add_argument('-E', '--exit-on-error', action='store_true',
+                        dest='errexit',
+                        help='Exit if an error is detected in the file analysis stages.')
     parser.add_argument('-H', '--overwrite-hpss', action='store_true',
                         dest='overwrite_hpss',
                         help='Ignore any existing HPSS cache files.')
@@ -721,7 +724,7 @@ def main():
     disk_roots = physical_disks(release_root, config)
     status = scan_disk(disk_roots, disk_files_cache,
                        overwrite=options.overwrite_disk)
-    if not status:
+    if options.errexit and not status:
         return 1
     #
     # See if the files are on HPSS.
@@ -732,7 +735,7 @@ def main():
     logger.debug("missing_files_cache = '%s'", missing_files_cache)
     status = find_missing(hpss_map, hpss_files, disk_files_cache,
                           missing_files_cache, options.report, options.limit)
-    if not status:
+    if options.errexit and not status:
         return 1
     #
     # Post process to generate HPSS commands

setup.cfg

@@ -52,10 +52,8 @@ relative_files = True
 source =
     hpsspy
 omit =
-    hpsspy/_version.py
     hpsspy/conftest.py
     hpsspy/test/*
-    */hpsspy/_version.py
     */hpsspy/conftest.py
     */hpsspy/test/*