Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem: AIP Verification fails for Zipped bag transfers containing .DS_Store files in objects directory #214

Closed
helrond opened this issue Sep 27, 2018 · 0 comments

Comments

Projects
None yet
5 participants
@helrond
Copy link

commented Sep 27, 2018

Expected behaviour
A Zipped bag transfer is selected and the transfer is processed successfully through transfer and ingest microservices and the AIP and DIP are stored.

Current behaviour
A Zipped bag transfer containing a file named .DS_Store in the data/objects/ directory is selected. The transfer is processed successfully through transfer microservices but fails at the Verify AIP microservice with the following output:

stdout

Running extraction command: atool --extract-to=/var/archivematica/sharedDirectory/tmp/ceaa5d3d-118a-40eb-9f54-c36f53b26c4f -V0 /var/archivematica/sharedDirectory/currentlyProcessing/3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f.7z

7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,2 CPUs)

Processing archive: /var/archivematica/sharedDirectory/currentlyProcessing/3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f.7z

Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/METS.ceaa5d3d-118a-40eb-9f54-c36f53b26c4f.xml
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/objects/submissionDocumentation/transfer-3b34198a-df0c-4c95-a374-b15dc6418cfa-5ba94856-e8b7-4641-ab69-67b10cf7f321/METS.xml
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/README.html
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/logs/transfers/3b34198a-df0c-4c95-a374-b15dc6418cfa-5ba94856-e8b7-4641-ab69-67b10cf7f321/logs/BagIt/bag-info.txt
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/bag-info.txt
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/logs/transfers/3b34198a-df0c-4c95-a374-b15dc6418cfa-5ba94856-e8b7-4641-ab69-67b10cf7f321/logs/BagIt/bagit.txt
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/bagit.txt
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/objects/metadata/transfers/3b34198a-df0c-4c95-a374-b15dc6418cfa-5ba94856-e8b7-4641-ab69-67b10cf7f321/manifest-md5.txt
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/manifest-sha256.txt
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/logs/transfers/3b34198a-df0c-4c95-a374-b15dc6418cfa-5ba94856-e8b7-4641-ab69-67b10cf7f321/logs/BagIt/tagmanifest-md5.txt
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/tagmanifest-md5.txt
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/objects/Lorem_ipsum_dolor_sit_amet.docx
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/objects/consectetur_adipiscing_elit.rtf
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/objects/consectetur_adipiscing_elit-794c1791-bd06-4d73-a081-641f6575cc92.pdf
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/objects/consectetur_adipiscing_elit.pdf
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/objects/Lorem_ipsum.odt
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/objects/metadata/transfers/3b34198a-df0c-4c95-a374-b15dc6418cfa-5ba94856-e8b7-4641-ab69-67b10cf7f321/rights.csv
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/objects/metadata.json
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/logs/transfers/3b34198a-df0c-4c95-a374-b15dc6418cfa-5ba94856-e8b7-4641-ab69-67b10cf7f321/logs/fileFormatIdentification.log
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/logs/fileFormatIdentification.log
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/logs/transfers/3b34198a-df0c-4c95-a374-b15dc6418cfa-5ba94856-e8b7-4641-ab69-67b10cf7f321/logs/filenameCleanup.log
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/logs/filenameCleanup.log
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/logs/removeUnneededFiles.log
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/objects/more_data.xlsx
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/objects/sample_data.xlsx
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/objects/submissionDocumentation/transfer-3b34198a-df0c-4c95-a374-b15dc6418cfa-5ba94856-e8b7-4641-ab69-67b10cf7f321
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/objects/submissionDocumentation
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/objects/metadata/transfers/3b34198a-df0c-4c95-a374-b15dc6418cfa-5ba94856-e8b7-4641-ab69-67b10cf7f321
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/objects/metadata/transfers
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/objects/metadata
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/objects
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/logs/transfers/3b34198a-df0c-4c95-a374-b15dc6418cfa-5ba94856-e8b7-4641-ab69-67b10cf7f321/logs/BagIt
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/logs/transfers/3b34198a-df0c-4c95-a374-b15dc6418cfa-5ba94856-e8b7-4641-ab69-67b10cf7f321/logs
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/logs/transfers/3b34198a-df0c-4c95-a374-b15dc6418cfa-5ba94856-e8b7-4641-ab69-67b10cf7f321
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/logs/transfers
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data/logs
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/data
Extracting  3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f

Everything is Ok

Folders: 13
Files: 25
Size:       930145
Compressed: 419126
Running test:  /usr/share/bagit/bin/bag verifyvalid "/var/archivematica/sharedDirectory/tmp/ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f"
Result is true.
Running test:  /usr/share/bagit/bin/bag checkpayloadoxum "/var/archivematica/sharedDirectory/tmp/ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f"
Payload-Oxum matches.
Running test:  /usr/share/bagit/bin/bag verifycomplete "/var/archivematica/sharedDirectory/tmp/ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f"
Result is true.
Running test:  /usr/share/bagit/bin/bag verifypayloadmanifests "/var/archivematica/sharedDirectory/tmp/ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f"
Result is true.
Running test:  /usr/share/bagit/bin/bag verifytagmanifests "/var/archivematica/sharedDirectory/tmp/ceaa5d3d-118a-40eb-9f54-c36f53b26c4f/3b34198a-df0c-4c95-a374-b15dc6418cfa-ceaa5d3d-118a-40eb-9f54-c36f53b26c4f"
Result is true.

stderr

'NoneType' object has no attribute 'startswith'Traceback (most recent call last):
  File "/src/MCPClient/lib/job.py", line 102, in JobContext
    yield
  File "/src/MCPClient/lib/clientScripts/verify_aip.py", line 271, in call
    job.set_status(verify_aip(job))
  File "/src/MCPClient/lib/clientScripts/verify_aip.py", line 247, in verify_aip
    verify_checksums(job, bag, sip_uuid)
  File "/src/MCPClient/lib/clientScripts/verify_aip.py", line 165, in verify_checksums
    if not file_.currentlocation.startswith('%SIPDirectory%objects/'):
AttributeError: 'NoneType' object has no attribute 'startswith'

Steps to reproduce

  1. Create a bag of a directory containing a .DS_Store file.
  2. Serialize the bag.
  3. Process the bag as described above.

There appears to be some discussion from the Archivematica forum on this, with Tim Walsh appearing to raise precisely the issue I'm experiencing.

Your environment (version of Archivematica, OS version, etc)
Archivematica v 1.8 qa/1.x branch (docker-compose)
Mac OS High Sierra


For Artefactual use:
Please make sure these steps are taken before moving this issue from Review to Verified in Waffle:

  • All PRs related to this issue are properly linked 👍
  • All PRs related to this issue have been merged 👍
  • Test plan for this issue has been implemented and passed 👍
  • Release documentation (e.g. release notes, wiki documentation, etc) regarding this issue has been written 👍

@helrond helrond changed the title Problem: AIP Verification fails for Zipped bag transfers containing .DSStore files in objects directory Problem: AIP Verification fails for Zipped bag transfers containing .DS_Store files in objects directory Sep 27, 2018

@nickwilkinson nickwilkinson added this to the 1.8.0 milestone Oct 1, 2018

@sevein sevein self-assigned this Oct 2, 2018

@sevein sevein removed their assignment Oct 2, 2018

@ablwr ablwr self-assigned this Oct 10, 2018

@sromkey sromkey closed this Nov 22, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.