Merge pull request #46 from zimeon/develop

Release 1.2.1

.travis.yml

@@ -1,21 +1,24 @@
-dist: xenial
+dist: bionic
 language: python
 python:
   - "3.6"
   - "3.7"
   - "3.8"
+  - "3.9"
 # Install sub-module using public HTTPS not SSH 
 git:
   submodules: false
 before_install:
   - sed -i 's/git@github.com:/https:\/\/github.com\//' .gitmodules
   - git submodule update --init --recursive
 install:
-  - pip install coveralls pycodestyle pep257
+  - pip install bagit  #pylint is unhappy with bagit install via setup.py, not sure why
+  - pip install coveralls pycodestyle pydocstyle pylint
   - python setup.py install
 script:
-  - pycodestyle --ignore=E501,W503 ocfl/*.py tests/*.py *.py
-  - pep257 ocfl/*.py tests/*.py *.py
+  - pycodestyle --ignore=E501,W503 *.py ocfl tests
+  - pydocstyle *.py ocfl tests
+  - pylint --disable=unneeded-not,line-too-long,unnecessary-semicolon,trailg-whitespace,missing-final-newline,bad-indentation,multiple-statements,bare-except,missing-module-docstring,missing-class-docstring,missing-function-docstring,W0511,W0622,W0707,C0103,R0902,R0911,R0912,R0913,R0914,R0915,R1702 *.py ocfl tests
   - python setup.py test
 after_success:
   - python setup.py coverage

CHANGES.md

@@ -1,5 +1,13 @@
 # ocfl-py changelog
 
+## 2021-04-18 v1.2.1
+
+  * Add use of `pylint` in addition to `pycodestyle` and `pydocstyle` (was `pep257`). Numerous minor fixes as a result of errors/warnings reported.
+  * Use additional fixtures in https://github.com/OCFL/fixtures for tests
+  * Validation:
+    * Correct missing root inventory from E034 to E063
+    * Add tests for digests in prior version manifests
+
 ## 2021-03-24 v1.2.0
 
   * Add ability for `ocfl-validate.py` to validate a standalone inventory
@@ -44,7 +52,7 @@
 ## 2020-05-18 v0.0.7
 
   * Validator now checks fixity block structure, additional fixity values in fixity block
-  * Validator now checks for repeated digests in manifest, fixity and state blocks (https://ocfl.io/1.0/spec/#E096, https://ocfl.io/1.0/spec/#E097, https://ocfl.io/1.0/spec/#E098) 
+  * Validator now checks for repeated digests in manifest, fixity and state blocks (https://ocfl.io/1.0/spec/#E096, https://ocfl.io/1.0/spec/#E097, https://ocfl.io/1.0/spec/#E098)
   * Move all the many README_*.md demos into docs folder
   * Add build_demo_docs.sh to build demo descriptions in docs folder
 
@@ -60,7 +68,7 @@
 
 ## 2020-05-05 v0.0.5
 
-  * Renumber errors to align somewhat with the canonical code set extracted 
+  * Renumber errors to align somewhat with the canonical code set extracted
     at https://github.com/OCFL/spec/blob/main/validation/validation-codes.md
   * Add --version parameter to scripts to show version number
 

CONTRIBUTING.md

@@ -12,10 +12,18 @@ If you propose a major change or new feature, please submit an issue to discuss
 
 If submitting a pull request:
 
-   * Understand that this code and any merged contributions are covered by the [MIT license](LICENSE.txt).
-   * Please discuss in a issue before submitting a pull request for significant changes.
-   * Please submit pull requests against the `main` branch (at this early stage of development changes are merged to `main`, this may change at some stage).
-   * Please follow [PEP8](https://www.python.org/dev/peps/pep-0008/) and [PEP257](https://www.python.org/dev/peps/pep-0257/) style rules. PEP8 line length (E501) is not enforced, just be reasonable.
-   * The warning W504 is enabled, but W503 is disables -- line breaks should occur before binary operators
-   * Please don't repeat code.
-   * Please cover the code with tests.
+   * Understand that this code and any merged contributions are covered by the [MIT license](LICENSE.txt)
+   * Please discuss in a issue before submitting a pull request for significant changes
+   * Please submit pull requests against the `develop` branch
+   * Please write code that passes the linting tests in `.travis.yml`, these include:
+     * `pycodestyle` implements [PEP8](https://www.python.org/dev/peps/pep-0008/) with the following warnings disabled:
+       * Line length is not enforced (E501), just be reasonable
+       * Where necessary, line breaks should occur before binary operators (warning W504 is enabled, but W503 is disabled)
+     * `pydocstyle` implements [PEP257](https://www.python.org/dev/peps/pep-0257/) style rules:
+       * Nothing is disabled
+     * `pylint` implements more complex static analysis and looks for code smells, some rules are disabled including:
+       * [Checks already implemented in `pycodestyle` and `pydocstyle`](http://pylint.pycqa.org/en/latest/faq.html#i-am-using-another-popular-linter-alongside-pylint-which-messages-should-i-disable-to-avoid-duplicates)
+       * A FIXME doesn't generate and error, but please avoid anyway (W0511)
+       * See `.travis.yml` for current exclusions
+   * Please don't repeat code
+   * Please cover the code with tests

docs/demo_build_spec_examples.md

@@ -10,8 +10,7 @@ The digest type sha512-spec-ex is sha512 with most of the content stripped out a
 
 ```
 > python ocfl-object.py --build --src fixtures/1.0/content/spec-ex-minimal --id http://example.org/minimal --digest sha512-spec-ex --created 2018-10-02T12:00:00Z --message One file --name Alice --address alice@example.org -v
-WARNING:ocfl.object:### Inventory for v1
-{
+WARNING:ocfl.object:### Inventory for v1{
   "digestAlgorithm": "sha512-spec-ex",
   "head": "v1",
   "id": "http://example.org/minimal",
@@ -37,6 +36,7 @@ WARNING:ocfl.object:### Inventory for v1
     }
   }
 }
+
 ```
 
 
@@ -48,8 +48,7 @@ This is inventory should match the example with 3 versions in <https://ocfl.io/1
 
 ```
 > python ocfl-object.py --build --src fixtures/1.0/content/spec-ex-full --id ark:/12345/bcd987 --fixity md5 --fixity sha1 --digest sha512-spec-ex -v
-WARNING:ocfl.object:### Inventory for v1
-{
+WARNING:ocfl.object:### Inventory for v1{
   "digestAlgorithm": "sha512-spec-ex",
   "fixity": {
     "md5": {
@@ -91,7 +90,7 @@ WARNING:ocfl.object:### Inventory for v1
   "type": "https://ocfl.io/1.0/spec/#inventory",
   "versions": {
     "v1": {
-      "created": "2021-03-24T20:22:33.307165Z",
+      "created": "2021-04-18T15:14:51.784558Z",
       "state": {
         "7dcc352f96c56dc...c31": [
           "foo/bar.xml"
@@ -106,8 +105,8 @@ WARNING:ocfl.object:### Inventory for v1
     }
   }
 }
-WARNING:ocfl.object:### Inventory for v2
-{
+
+WARNING:ocfl.object:### Inventory for v2{
   "digestAlgorithm": "sha512-spec-ex",
   "fixity": {
     "md5": {
@@ -158,7 +157,7 @@ WARNING:ocfl.object:### Inventory for v2
   "type": "https://ocfl.io/1.0/spec/#inventory",
   "versions": {
     "v1": {
-      "created": "2021-03-24T20:22:33.307165Z",
+      "created": "2021-04-18T15:14:51.784558Z",
       "state": {
         "7dcc352f96c56dc...c31": [
           "foo/bar.xml"
@@ -172,7 +171,7 @@ WARNING:ocfl.object:### Inventory for v2
       }
     },
     "v2": {
-      "created": "2021-03-24T20:22:33.308961Z",
+      "created": "2021-04-18T15:14:51.786202Z",
       "state": {
         "4d27c86b026ff70...b53": [
           "foo/bar.xml"
@@ -185,8 +184,8 @@ WARNING:ocfl.object:### Inventory for v2
     }
   }
 }
-WARNING:ocfl.object:### Inventory for v3
-{
+
+WARNING:ocfl.object:### Inventory for v3{
   "digestAlgorithm": "sha512-spec-ex",
   "fixity": {
     "md5": {
@@ -237,7 +236,7 @@ WARNING:ocfl.object:### Inventory for v3
   "type": "https://ocfl.io/1.0/spec/#inventory",
   "versions": {
     "v1": {
-      "created": "2021-03-24T20:22:33.307165Z",
+      "created": "2021-04-18T15:14:51.784558Z",
       "state": {
         "7dcc352f96c56dc...c31": [
           "foo/bar.xml"
@@ -251,7 +250,7 @@ WARNING:ocfl.object:### Inventory for v3
       }
     },
     "v2": {
-      "created": "2021-03-24T20:22:33.308961Z",
+      "created": "2021-04-18T15:14:51.786202Z",
       "state": {
         "4d27c86b026ff70...b53": [
           "foo/bar.xml"
@@ -263,7 +262,7 @@ WARNING:ocfl.object:### Inventory for v3
       }
     },
     "v3": {
-      "created": "2021-03-24T20:22:33.310220Z",
+      "created": "2021-04-18T15:14:51.787447Z",
       "state": {
         "4d27c86b026ff70...b53": [
           "foo/bar.xml"
@@ -278,6 +277,7 @@ WARNING:ocfl.object:### Inventory for v3
     }
   }
 }
+
 ```
 
 
@@ -289,8 +289,7 @@ This is inventory should match the example showing how content paths may differ
 
 ```
 > python ocfl-object.py --build --src fixtures/1.0/content/spec-ex-diff-paths --id http://example.org/diff-paths --digest sha512-spec-ex --normalization md5 --created 2019-03-14T20:31:00Z -v
-WARNING:ocfl.object:### Inventory for v1
-{
+WARNING:ocfl.object:### Inventory for v1{
   "digestAlgorithm": "sha512-spec-ex",
   "head": "v1",
   "id": "http://example.org/diff-paths",
@@ -317,5 +316,6 @@ WARNING:ocfl.object:### Inventory for v1
     }
   }
 }
+
 ```
 

docs/demo_ocfl_object_script.md

@@ -10,7 +10,7 @@ The `--version` argument will show version number and exit
 
 ```
 > python ocfl-object.py --version
-ocfl-object.py is part of ocfl-py version 1.2.0
+ocfl-object.py is part of ocfl-py version 1.2.1
 ```
 
 
@@ -22,8 +22,7 @@ Without an `--objdir` argument the script just writes out the inventory for the
 
 ```
 > python ocfl-object.py --create --id http://example.org/obj1 --src fixtures/1.0/content/cf1/v1
-WARNING:ocfl.object:### Inventory for v1
-{
+WARNING:ocfl.object:### Inventory for v1{
   "digestAlgorithm": "sha512",
   "head": "v1",
   "id": "http://example.org/obj1",
@@ -35,7 +34,7 @@ WARNING:ocfl.object:### Inventory for v1
   "type": "https://ocfl.io/1.0/spec/#inventory",
   "versions": {
     "v1": {
-      "created": "2021-03-24T20:22:34.576958Z",
+      "created": "2021-04-18T15:14:53.278161Z",
       "state": {
         "43a43fe8a8a082d3b5343dfaf2fd0c8b8e370675b1f376e92e9994612c33ea255b11298269d72f797399ebb94edeefe53df243643676548f584fb8603ca53a0f": [
           "a_file.txt"
@@ -44,6 +43,7 @@ WARNING:ocfl.object:### Inventory for v1
     }
   }
 }
+
 ```
 
 
@@ -53,8 +53,7 @@ Without an `--objdir` argument the script just writes out the inventory for each
 
 ```
 > python ocfl-object.py --build --id http://example.org/obj2 --src fixtures/1.0/content/cf3
-WARNING:ocfl.object:### Inventory for v1
-{
+WARNING:ocfl.object:### Inventory for v1{
   "digestAlgorithm": "sha512",
   "head": "v1",
   "id": "http://example.org/obj2",
@@ -66,7 +65,7 @@ WARNING:ocfl.object:### Inventory for v1
   "type": "https://ocfl.io/1.0/spec/#inventory",
   "versions": {
     "v1": {
-      "created": "2021-03-24T20:22:34.974946Z",
+      "created": "2021-04-18T15:14:53.721542Z",
       "state": {
         "43a43fe8a8a082d3b5343dfaf2fd0c8b8e370675b1f376e92e9994612c33ea255b11298269d72f797399ebb94edeefe53df243643676548f584fb8603ca53a0f": [
           "a_file.txt"
@@ -75,8 +74,8 @@ WARNING:ocfl.object:### Inventory for v1
     }
   }
 }
-WARNING:ocfl.object:### Inventory for v2
-{
+
+WARNING:ocfl.object:### Inventory for v2{
   "digestAlgorithm": "sha512",
   "head": "v2",
   "id": "http://example.org/obj2",
@@ -91,15 +90,15 @@ WARNING:ocfl.object:### Inventory for v2
   "type": "https://ocfl.io/1.0/spec/#inventory",
   "versions": {
     "v1": {
-      "created": "2021-03-24T20:22:34.974946Z",
+      "created": "2021-04-18T15:14:53.721542Z",
       "state": {
         "43a43fe8a8a082d3b5343dfaf2fd0c8b8e370675b1f376e92e9994612c33ea255b11298269d72f797399ebb94edeefe53df243643676548f584fb8603ca53a0f": [
           "a_file.txt"
         ]
       }
     },
     "v2": {
-      "created": "2021-03-24T20:22:34.975605Z",
+      "created": "2021-04-18T15:14:53.722350Z",
       "state": {
         "296e72b8fd5f7f0ac1473993600ae34953d5dab646f17e7b182b8648aff830d7bf01b56490777cb3e72b33fcc1ae520506badea1032252d1a55fd7362e269975": [
           "a_file.txt"
@@ -108,8 +107,8 @@ WARNING:ocfl.object:### Inventory for v2
     }
   }
 }
-WARNING:ocfl.object:### Inventory for v3
-{
+
+WARNING:ocfl.object:### Inventory for v3{
   "digestAlgorithm": "sha512",
   "head": "v3",
   "id": "http://example.org/obj2",
@@ -124,23 +123,23 @@ WARNING:ocfl.object:### Inventory for v3
   "type": "https://ocfl.io/1.0/spec/#inventory",
   "versions": {
     "v1": {
-      "created": "2021-03-24T20:22:34.974946Z",
+      "created": "2021-04-18T15:14:53.721542Z",
       "state": {
         "43a43fe8a8a082d3b5343dfaf2fd0c8b8e370675b1f376e92e9994612c33ea255b11298269d72f797399ebb94edeefe53df243643676548f584fb8603ca53a0f": [
           "a_file.txt"
         ]
       }
     },
     "v2": {
-      "created": "2021-03-24T20:22:34.975605Z",
+      "created": "2021-04-18T15:14:53.722350Z",
       "state": {
         "296e72b8fd5f7f0ac1473993600ae34953d5dab646f17e7b182b8648aff830d7bf01b56490777cb3e72b33fcc1ae520506badea1032252d1a55fd7362e269975": [
           "a_file.txt"
         ]
       }
     },
     "v3": {
-      "created": "2021-03-24T20:22:34.976141Z",
+      "created": "2021-04-18T15:14:53.723184Z",
       "state": {
         "43a43fe8a8a082d3b5343dfaf2fd0c8b8e370675b1f376e92e9994612c33ea255b11298269d72f797399ebb94edeefe53df243643676548f584fb8603ca53a0f": [
           "a_file.txt"
@@ -149,6 +148,7 @@ WARNING:ocfl.object:### Inventory for v3
     }
   }
 }
+
 ```
 
 

docs/demo_ocfl_sidecar_script.md

@@ -10,7 +10,7 @@ The `--version` argument will show version number and exit
 
 ```
 > python ocfl-sidecar.py --version
-ocfl-sidecar.py is part of ocfl-py version 1.2.0
+ocfl-sidecar.py is part of ocfl-py version 1.2.1
 ```
 
 

docs/demo_ocfl_store_script.md

@@ -10,7 +10,7 @@ The `--version` argument will show version number and exit (but we still tave to
 
 ```
 > python ocfl-store.py --version --root=tmp/root --list
-ocfl-store.py is part of ocfl-py version 1.2.0
+ocfl-store.py is part of ocfl-py version 1.2.1
 ```
 
 

docs/demo_using_bagit_bags.md

@@ -161,8 +161,8 @@ Taking the newly created OCFL object `/tmp/obj` we can `--extract` the `v4` cont
 INFO:ocfl.object:Extracted v4 into tmp/extracted_v4
 INFO:bagit:Creating bag for directory tmp/extracted_v4
 INFO:bagit:Creating data directory
-INFO:bagit:Moving my_content to /privatetmp/extracted_v4/tmpmv9q6tbn/my_content
-INFO:bagit:Moving /privatetmp/extracted_v4/tmpmv9q6tbn to data
+INFO:bagit:Moving my_content to /privatetmp/extracted_v4/tmp1z_q233f/my_content
+INFO:bagit:Moving /privatetmp/extracted_v4/tmp1z_q233f to data
 INFO:bagit:Using 1 processes to generate manifests: sha512
 INFO:bagit:Generating manifest lines for file data/my_content/dracula.txt
 INFO:bagit:Generating manifest lines for file data/my_content/dunwich.txt
@@ -184,12 +184,12 @@ We note that the OCFL object had only one `content` file in `v4` but the extract
 diff -r tmp/extracted_v4/bag-info.txt tests/testdata/bags/uaa_v4/bag-info.txt
 1,2c1
 < Bag-Software-Agent: bagit.py v1.8.1 <https://github.com/LibraryOfCongress/bagit-python>
-< Bagging-Date: 2021-03-24
+< Bagging-Date: 2021-04-18
 ---
 > Bagging-Date: 2020-01-04
 diff -r tmp/extracted_v4/tagmanifest-sha512.txt tests/testdata/bags/uaa_v4/tagmanifest-sha512.txt
 2c2
-< 4354c74d41f895abfcb617379a7a4114dcff7193a8f30ae27f8d5553fdecce4c533cfdf6f8f94ad2ab15adff22f0e5768ee8961f5ac6aa5156c79ae8a6642651 bag-info.txt
+< b28abe8d128a679ebfa949ac900b4f63119d5734ab4cf60c102aacdf69f486cb69ff143a3f95d0be9d0d942ee7a69b7587d72abf61d2c281dc10d9f62b1b0339 bag-info.txt
 ---
 > 10624e6d45462def7af66d1a0d977606c7b073b01809c1d42258cfab5c34a275480943cbe78044416aee1f23822cc3762f92247b8f39b5c6ddc5ae32a8f94ce5 bag-info.txt
 ```