Fixity is a command line tool that assists in checking fixity for AIPs stored in the Archivematica Storage Service.
"Fixity" in digital preservation terms is defined as "the assurance that a digital file has remained unchanged, i.e. fixed." (Bailey, 2014) and more information about that, and the concept of checksums, can be found in the Digital Preservation Coalition's (DPC) Digital Preservation Handbook.
Fixity is a client application that calls the Storage Service's Check Fixity API endpoint for a single AIP or for each AIP in the Storage Service.
The Storage Service's Check Fixity endpoint can be used to trigger a fixity check on an individual package. When the Fixity client calls this endpoint, the checking is done in the Storage Service itself.
Fixity aggregates results and makes it easy to do this across all packages.
To trigger fixity checking for a package, we need to know the AIP UUID and the Storage Service authorization parameters.
The URL is constructed as follows:
http://127.0.0.1:62081/api/v2/file/<AIP UUID>/check_fixity/
And we can supply authentication information in a HTTP request using a command line tool such as HTTPie:
http -v --pretty=format \
GET "http://127.0.0.1:62081/api/v2/file/5e21dd0d-190e-4ffb-b752-76d860bea898/check_fixity/" \
Authorization:"ApiKey test:test"
This results in the following JSON output showing us that the Fixity check was a 'success', that is, there were no errors:
{
"failures": {
"files": {
"changed": [],
"missing": [],
"untracked": []
}
},
"message": "",
"success": true,
"timestamp": null
}
This detailed JSON output is recorded in Fixity's internal database. Summary
information about errors is printed to the console. The output for package
5e21dd0d-190e-4ffb-b752-76d860bea898
would simply look as follows:
Fixity scan succeeded for AIP: 5e21dd0d-190e-4ffb-b752-76d860bea898
The Check Fixity endpoint updates the Storage Service fixitylog
model with
details of the last check completed.
Storage Service users can see the last Fixity Date
and Fixity Status
in the
Storage Service Packages panel by going to {storage-service-url}/packages/
Administrators of the Storage Service can query the Storage Service database as follows:
select
package_id as "AIP UUID",
datetime_reported as "Fixity Last Checked",
success,
error_details
from locations_fixitylog
where package_id = '5e21dd0d-190e-4ffb-b752-76d860bea898'
order by datetime_reported desc
limit 1;
And we can see the details of the check in the SQL output:
+--------------------------------------+----------------------------+---------+---------------+
| AIP UUID | Fixity Last Checked | success | error_details |
+--------------------------------------+----------------------------+---------+---------------+
| 5e21dd0d-190e-4ffb-b752-76d860bea898 | 2018-11-06 11:20:43.634188 | 1 | NULL |
+--------------------------------------+----------------------------+---------+---------------+
The Storage Service can detect multiple categories of error. Some are shown as outputs of Fixity below with the corresponding detailed information that can be accessed from Fixity's database:
Fixity scan failed for AIP: 5e21dd0d-190e-4ffb-b752-76d860bea898 (Oxum error.
Found 10 files and 126404 bytes on disk; expected 10 files and 126405 bytes.)
Detail:
{
"failures": {
"files": {
"changed": [],
"missing": [],
"untracked": []
}
},
"finished": 1541505247,
"message": "Oxum error. Found 10 files and 126404 bytes on disk; expected 10 files and 126405 bytes.",
"started": 1541505247,
"success": false,
"timestamp": null
}
Fixity scan failed for AIP: 5e21dd0d-190e-4ffb-b752-76d860bea898 (invalid bag)
Detail:
{
"failures": {
"files": {
"changed": [
{
"actual": "6ea7859a4edb8406aae0dfdf5abda9db97eeb8b416922b7356082ab55c9e4161",
"expected": "9fd57a8c09e0443de485cf51b68ad8ef54486454434daed499d2f686b7efc2b4",
"hash_type": "sha256",
"message": "data/objects/one_file/one_file.txt checksum validation failed (alg=sha256 expected=9fd57a8c09e0443de485cf51b68ad8ef54486454434daed499d2f686b7efc2b4 found=6ea7859a4edb8406aae0dfdf5abda9db97eeb8b416922b7356082ab55c9e4161)",
"path": "data/objects/one_file/one_file.txt"
}
],
"missing": [],
"untracked": []
}
},
"finished": 1541505387,
"message": "invalid bag",
"started": 1541505386,
"success": false,
"timestamp": null
}
Fixity scan failed for AIP: 5e21dd0d-190e-4ffb-b752-76d860bea898 (Oxum error.
Found 9 files and 126386 bytes on disk; expected 10 files and 126405 bytes.)
Detail:
{
"failures": {
"files": {
"changed": [],
"missing": [],
"untracked": []
}
},
"finished": 1541505645,
"message": "Oxum error. Found 9 files and 126386 bytes on disk; expected 10 files and 126405 bytes.",
"started": 1541505645,
"success": false,
"timestamp": null
}
Fixity scan failed for AIP: 5e21dd0d-190e-4ffb-b752-76d860bea898 (Oxum error.
Found 11 files and 126405 bytes on disk; expected 10 files and 126405 bytes.)
Detail:
{
"failures": {
"files": {
"changed": [],
"missing": [],
"untracked": []
}
},
"finished": 1541505549,
"message": "Oxum error. Found 11 files and 126405 bytes on disk; expected 10 files and 126405 bytes.",
"started": 1541505549,
"success": false,
"timestamp": null
}
Fixity scan failed for AIP: 5e21dd0d-190e-4ffb-b752-76d860bea898 (Missing
manifest file)
Detail:
{
"failures": {
"files": {
"changed": [],
"missing": [],
"untracked": []
}
},
"finished": 1541505815,
"message": "Missing manifest file",
"started": 1541505815,
"success": false,
"timestamp": null
}
For any error in the Fixity check the Storage Service database maintains a summary log, see for example when data in a file has been modified:
+--------------------------------------+----------------------------+---------+---------------+
| AIP UUID | Fixity Last Checked | success | error_details |
+--------------------------------------+----------------------------+---------+---------------+
| 5e21dd0d-190e-4ffb-b752-76d860bea898 | 2018-11-06 12:10:22.981609 | 0 | invalid bag |
+--------------------------------------+----------------------------+---------+---------------+
Installation of Fixity can be completed with the following steps:
-
Checkout or link the code to
/usr/lib/archivematica/fixity
-
Go to
/usr/lib/archivematica/
user@root:~$ cd /usr/lib/archivematica/
-
Clone the code:
user@root:/usr/lib/archivematica/$ git clone https://github.com/artefactual/fixity.git
Once this is complete, the directory
/usr/lib/archivematica/fixity
should exist.cd
back to the home directoryuser@root:/usr/lib/archivematica/$ cd
-
-
Create a virtualenv in
/usr/share/python/fixity
, and install fixity and dependencies in it-
Switch to root
user@root:~$ sudo -i
-
Run:
root@host:~# virtualenv /usr/share/python/fixity root@host:~# source /usr/share/python/fixity/bin/activate (fixity)root@host:~# cd /usr/lib/archivematica/fixity (fixity)root@host:/usr/lib/archivematica/fixity# pip install -r requirements.txt (fixity)root@host:/usr/lib/archivematica/fixity# pip install -e .
-
-
Create a symlink from the executable to
/usr/local/bin
. You must still be root.(fixity)root@host:/usr/lib/archivematica/fixity# ln -s /usr/share/python/fixity/bin/fixity /usr/local/bin/fixity
-
Export required environment variables. For ease of use later, creating
/etc/profile.d/fixity.sh
is recommended:-
To create the file:
(fixity)root@host:/usr/lib/archivematica/fixity# touch /etc/profile.d/fixity.sh (fixity)root@host:/usr/lib/archivematica/fixity# nano /etc/profile.d/fixity.sh
-
You are now editing the environment variables file. You should use the URL of your Storage Service, and the username and API key of one Storage Service user. Replace the URL, user and key with your data.
#!/bin/bash export STORAGE_SERVICE_URL=http://localhost:8000 export STORAGE_SERVICE_USER=myuser export STORAGE_SERVICE_KEY=myapikey
-
Optionally, if you are using Fixity with a reporting service, you can also add:
export REPORT_URL=http://myurl.com export REPORT_USERNAME=myuser export REPORT_PASSWORD=mypassword
-
Load the variables from the file.
(fixity)root@host:/usr/lib/archivematica/fixity# source /etc/profile.d/fixity.sh
-
-
Run the tool with sudo or as root the first time. Subsequent runs can be with any user.
(fixity)root@host:/usr/lib/archivematica/fixity# fixity scanall
-
To exit the virtualenv:
(fixity)root@host:/usr/lib/archivematica/fixity# deactivate root@host:/usr/lib/archivematica/fixity#
And to exit the root user:
root@host:/usr/lib/archivematica/fixity# exit user@host:~$
-
After the initial install, to run fixity you only need to load the variables you defined earlier and run fixity.
user@host:~$ source /etc/profile.d/fixity.sh user@host:~$ fixity scanall
For more information on usage, consult the manpage.
To set up this repository for local development:
-
Clone this repository:
git clone git@github.com:artefactual/fixity
-
Enter the repository directory:
cd fixity
-
Create a virtual environment with a recent version of Python:
python3 -m venv .venv
-
Activate the virtual environment:
source .venv/bin/activate
-
Install the project in editable mode passing the development extra:
pip install -e .[dev]
This project uses tox
to manage and run tests with pytest
. You can find
the configuration details in the [tool.tox]
section of the pyproject.toml
file.
You can install tox
in your virtual environment:
pip install tox
Run all the tests this way:
tox -e py
You can pass options to pytest
:
tox -e py -- -vvv -k "test_fixity"
If you have a security concern about Archivematica or any of its companion repositories, please see the Archivematica security policy for information on how to safely report a vulnerability.
Fixity is copyright 2014-2018 Artefactual Systems Inc.