Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem: can't store encrypted uncompressed AIPs #294

Closed
sevein opened this issue Oct 20, 2018 · 11 comments

Comments

Projects
None yet
6 participants
@sevein
Copy link
Contributor

commented Oct 20, 2018

Expected behaviour
It should be possible to store encrypted uncompressed AIPS.

Current behaviour
Store the AIP fails:

STDOUT >>>
"/api/v2/location/2585ce71-a747-466b-b0ae-dffb4ebfd54f/" "/var/archivematica/sharedDirectory/currentlyProcessing/BagTransfer_1540057000-6b570ab1-072f-48cf-bed7-33320c5a273a/BagTransfer_1540057000-6b570ab1-072f-48cf-bed7-33320c5a273a" "6b570ab1-072f-48cf-bed7-33320c5a273a" "BagTransfer_1540057000" "SIP"
STDERR >>>
storeaip_v0.0: WARNING   2018-10-20 17:38:55,612  archivematica.mcp.client.storeAIP.store_aip:215  SIP creation failed: Failure storing file: <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'get_file'.
SIP creation failed: Failure storing file: <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'get_file'. See logs for more details.Traceback (most recent call last):
  File "/src/MCPClient/lib/job.py", line 102, in JobContext
    yield
  File "/src/MCPClient/lib/clientScripts/store_aip.py", line 313, in call
    args.sip_uuid, args.sip_name, args.sip_type))
  File "/src/MCPClient/lib/clientScripts/store_aip.py", line 216, in store_aip
    raise Exception(errmsg + " See logs for more details.")
Exception: SIP creation failed: Failure storing file: <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'get_file'. See logs for more details.

In Storage Service:

INFO      2018-10-20 10:38:53  locations.models.gpg:gpg:_create_tar:308:  creating archive of BagTransfer_1540057000-6b570ab1-072f-48cf-bed7-33320c5a273a at /var/archivematica/sharedDirectory/www/AIPsStoreEncrypted/6b57/0ab1/072f/48cf/bed7/3332/0c5a/273a/BagTransfer_1540057000-6b570ab1-072f-48cf-bed7-33320c5a273a.tar, relative to /var/archivematica/sharedDirectory/www/AIPsStoreEncrypted/6b57/0ab1/072f/48cf/bed7/3332/0c5a/273a
INFO      2018-10-20 10:38:53  locations.models.gpg:gpg:_gpg_encrypt:239:  Successfully encrypted /var/archivematica/sharedDirectory/www/AIPsStoreEncrypted/6b57/0ab1/072f/48cf/bed7/3332/0c5a/273a/BagTransfer_1540057000-6b570ab1-072f-48cf-bed7-33320c5a273a at /var/archivematica/sharedDirectory/www/AIPsStoreEncrypted/6b57/0ab1/072f/48cf/bed7/3332/0c5a/273a/BagTransfer_1540057000-6b570ab1-072f-48cf-bed7-33320c5a273a.gpg
ERROR     2018-10-20 10:38:53  locations.models.async_manager:async_manager:wrapper:122:  Task threw an error: 'NoneType' object has no attribute 'get_file'
Traceback (most recent call last):
  File "/src/storage_service/locations/models/async_manager.py", line 119, in wrapper
    value = task_fn(*args, **kwargs)
  File "/src/storage_service/locations/api/resources.py", line 701, in task
    self._store_bundle(bundle)
  File "/src/storage_service/locations/api/resources.py", line 675, in _store_bundle
    premis_agents=agents, aip_subtype=aip_subtype)
  File "/src/storage_service/locations/models/package.py", line 613, in store_aip
    pointer_file, storage_effects))
  File "/src/storage_service/locations/models/package.py", line 994, in create_new_pointer_file_given_storage_effects
    old_fsentry = old_pointer_file.get_file(file_uuid=self.uuid)
AttributeError: 'NoneType' object has no attribute 'get_file'

Steps to reproduce
From aip-encryption.feature:

  @uncompressed
  Scenario: Richard wants to ensure that he can encrypt uncompressed AIPs.
    Given there is a standard GPG-encrypted space in the storage service
    And there is a standard GPG-encrypted AIP Storage location in the storage service
    And the default processing config is in its default state
    And the processing config decision "Select compression algorithm" is set to "Uncompressed"
    When a transfer is initiated on directory ~/archivematica-sampledata/SampleTransfers/BagTransfer
    And standard AIP-creation decisions are made
    And the user waits for the "Store AIP location" decision point to appear and chooses "Store AIP Encrypted in standard Archivematica Directory" during ingest
    And the user waits for the AIP to appear in archival storage
    And the user queries the API until the AIP has been stored
    Then the uncompressed AIP on disk at /var/archivematica/sharedDirectory/www/AIPsStoreEncrypted/ is encrypted
    When the user downloads the AIP
    Then the downloaded uncompressed AIP is an unencrypted tarfile

~/archivematica-sampledata/SampleTransfers/BagTransfer can be found at the sampledata repository.

Your environment (version of Archivematica, OS version, etc)
AM18 qa/1.x 89dfc80d


For Artefactual use:
Please make sure these steps are taken before moving this issue from Review to Verified in Waffle:

  • All PRs related to this issue are properly linked 👍
  • All PRs related to this issue have been merged 👍
  • Test plan for this issue has been implemented and passed 👍
  • Documentation regarding this issue has been written and it has been added to the release notes, if needed 👍

@sevein sevein added the Type: bug label Oct 20, 2018

@sevein sevein referenced this issue Oct 20, 2018

Merged

More fixes for AM18 #132

3 of 3 tasks complete

@nickwilkinson nickwilkinson added this to the 1.8.0 milestone Oct 22, 2018

@sevein

This comment has been minimized.

Copy link
Contributor Author

commented Oct 22, 2018

Could this issue be related?
artefactual/archivematica-storage-service#324

@ross-spencer

This comment has been minimized.

Copy link
Member

commented Oct 23, 2018

@sevein @jraddaoui my first instinct was pointer files. This was my last commit related to them, I wonder if this would pass before that? artefactual/archivematica-storage-service@c27fb48

@jraddaoui

This comment has been minimized.

Copy link

commented Oct 30, 2018

Hi @sevein, @ross-spencer, @joel-simpson,

artefactual/archivematica-storage-service#424 fixes the issue for now, just avoiding to access pointer files that may not exist, like in uncompressed AIPs. It should also fix similar issues in the replication and re-ingest processes.

Please, let me know if you think this is good enough or if you want me to tackle artefactual/archivematica-storage-service#324 for the 1.8 release.

@sevein

This comment has been minimized.

Copy link
Contributor Author

commented Oct 30, 2018

@jraddaoui, why this was not a problem when we tested 1.7 releases? I'm assuming we've made a change that is causing this? Was it introduced in SS v0.12? SS v0.13?

@ross-spencer

This comment has been minimized.

Copy link
Member

commented Oct 30, 2018

@jraddaoui @sevein knowing the regression would be useful I agree. For your other question though @jraddaoui (artefactual/archivematica-storage-service#324) @sallain or @sromkey might have a better idea why it didn't have a 1.8 milestone, but I assume, potential complexity, and sponsorship are two likely factors. Your fix seems like a nice in-between at this stage in the release for 1.8.

@jraddaoui

This comment has been minimized.

Copy link

commented Oct 30, 2018

@sevein, @ross-spencer,

Found this artefactual/archivematica-storage-service#350 linked in artefactual/archivematica-storage-service#324. I guess we never cherry-picked the fix to the qa branch.

@jraddaoui

This comment has been minimized.

Copy link

commented Oct 30, 2018

Fixed in SS repository, qa/0.x and stable/0.12.x branches.

@jraddaoui jraddaoui removed their assignment Oct 30, 2018

@sallain sallain self-assigned this Nov 1, 2018

@sallain

This comment has been minimized.

Copy link
Contributor

commented Nov 1, 2018

I am maybe 75% confident that this is resolved. @jraddaoui, can you check the settings on am18bionic to see if it's set up properly to be receiving encrypted AIPs? If everything looks good to you, then we know that it's working.

@jraddaoui

This comment has been minimized.

Copy link

commented Nov 2, 2018

Hi @sallain, lol-3 and lol-4 seem to be stored encrypted and uncompressed.

@sallain

This comment has been minimized.

Copy link
Contributor

commented Nov 2, 2018

Thank you @jraddaoui!!!

@sallain

This comment has been minimized.

Copy link
Contributor

commented Nov 2, 2018

Added to release notes; no docs needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.