Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem: cannot store AIP in DSpace due to file extension returned #69

Closed
sromkey opened this issue Aug 10, 2018 · 13 comments

Comments

Projects
None yet
5 participants
@sromkey
Copy link

commented Aug 10, 2018

Expected behaviour

You should be able to store an AIP in DSpace. The AIP is chunked into 2 parts- one for the objects and one that contains all the metadata, logs, etc.

Current behaviour

When you attempt to store an AIP in DSpace it will deposit successfully, but the Storage Service is returning an error because of DSpace returning an extension for one of the AIP chunks as .atom. In this function: https://github.com/artefactual/archivematica-storage-service/blob/5a94ce5e8a19be6fff22848bd366f6d6eb2c731d/storage_service/locations/models/package.py#L1210-L1247 the SS is expecting either the extension .7z or .bz2. Note that the DSpace space in the SS is "special-" you can set it to make zip files from your AIP chunks instead of 7z or bz2.

The Dashboard fails in Store the AIP job and the SS logs contain this error:
https://gist.github.com/sromkey/c53ebf8557791a48bcab0c7f6c1d6df9

Steps to reproduce

To store an AIP in DSpace, ensure you have configured the location/space correctly and also it is expecting several pieces of DC metadata (I forget exactly which ones so I've been filling out the whole DC template in testing).

Your environment (version of Archivematica, OS version, etc)

Archivematica qa deployed on a VM, with Storage Service 0.12 release candidate.

@sevein

This comment has been minimized.

Copy link
Contributor

commented Oct 10, 2018

How to reproduce and QA:

Set up DSpace Space

  1. Protocol: DSpace via SWORD2 API
  2. Path: leave it empty
  3. Staging Path: /var/archivematica/storage_service_dspace
  4. Service Document IRI: http://hostname:8080/swordv2/servicedocument
  5. Don't forget username and password!

Make sure that /var/archivematica/storage_service_dspace exists and it's writable by the archivematica user.

Add location AIP Storage

  1. Purpose: AIP Storage
  2. Pipeline: select the one needed
  3. Relative Path, e.g.: http://hostname:8080/swordv2/collection/123456789/2
    Make sure that it's a DSpace collection.
  4. Description: give it a name!

Start the transfer!

  1. Uncheck box Approve automatically
  2. Start a new standard transfer - I've used sampledeta's SampleTransfers/Images/pictures
  3. Before approving it, add metadata - fields required are: (otherwise the upload will fail)
    • dc:title
    • dc:description
    • dc:creator
    • dc:date
    • dc:rights
    • dc:relation
  4. Approve!
  5. In "Store AIP location" choose the location added above.

Expected results

  • "Store the AIP" should complete successfully.
  • The package is listed in Storage Service, although the value in "Current location" seems abnormal.
  • In "DSpace Home" (e.g. http://hostname:8080/xmlui/), I see the new item added under "Recently added". I can also find it under the collection.

qubot pushed a commit to artefactual/archivematica-storage-service that referenced this issue Oct 10, 2018

Store AIP: fix pointer file creation if extension is unknown
Store AIP was raising when the value in `current_path` had an unknown
extension. The code responsible for creating the `premis:object` element
in the pointer file was assuming that the extension was either `.7z` or
`.bz2`. However when the space is using DSpace, the path does not include
the extension. This commit changes the lookup so `format_name` and
`format_registry_key` are only populated when the extension is known.

Connects to archivematica/Issues#69.

@sevein sevein removed their assignment Oct 11, 2018

@sallain

This comment has been minimized.

Copy link
Contributor

commented Oct 12, 2018

@currmie Assigned to you for review as it can hopefully fit in with your other DSpace testing!

@currmie

This comment has been minimized.

Copy link

commented Oct 17, 2018

@sevein - test continuously fails during Store AIP. Keep getting error that looks like this:
OSError(13, 'Permission denied')
Error extracting AIP at "/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50//api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/%/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/S/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/I/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/P/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/D/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/i/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/r/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/e/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/c/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/t/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/o/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/r/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/y/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/%/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/%/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/A/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/I/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/P/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/F/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/i/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/l/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/e/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/n/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/a/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/m/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/e/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/%/api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50//api/v2/location/b09e95af-a4d6-4011-b901-09b8a5f0bf50/"

@currmie

This comment has been minimized.

Copy link

commented Oct 18, 2018

@sevein - I've also tested locally on my Compose environment and checked the logs:

archivematica-storage-service_1  | ERROR     2018-10-17 23:15:52  locations.models.async_manager:async_manager:wrapper:122:  Task threw an error: [Errno 13] Permission denied: '/var/archivematica/storage_service_dspace'
archivematica-storage-service_1  | OSError: [Errno 13] Permission denied: '/var/archivematica/storage_service_dspace' 

Seems like I need to make /var/archivematica/storage_service_dspace writable ?

@sevein

This comment has been minimized.

Copy link
Contributor

commented Oct 19, 2018

This is how you can create it:

$ docker-compose exec --user=root archivematica-storage-service bash
$ mkdir -p /var/archivematica/storage_service_dspace
$ chown -R archivematica:archivematica /var/archivematica/storage_service_dspace
@currmie

This comment has been minimized.

Copy link

commented Oct 19, 2018

So I created and made that directory writable, then ran a new transfer per your specifications in comments ^.

Now getting this cryptic error in the logs...

2018-10-18 18:12:03 locations.models.async_manager:async_manager:wrapper:122: Task threw an error: [Errno 111] Connection refused archivematica-storage-service_1 | raise socket.error, msg archivematica-storage-service_1 | error: [Errno 111] Connection refused

@sevein

This comment has been minimized.

Copy link
Contributor

commented Oct 19, 2018

It's not able to connect to the server, maybe a firewall is blocking the traffic from your IP address.
CC @mamedin @scollazo

@currmie

This comment has been minimized.

Copy link

commented Oct 19, 2018

Quick update: I am able to get the AIP staged on the SS locally, but I can't get the SS to communicate with the dspace5x server in order to verify what happens after the AIP is deposited successfully.

@mamedin

This comment has been minimized.

Copy link

commented Oct 19, 2018

There's no firewall at dspace5x server, but we can check the connection this morning.

@mamedin mamedin closed this Oct 19, 2018

@mamedin mamedin removed the Status: review label Oct 19, 2018

@currmie

This comment has been minimized.

Copy link

commented Oct 19, 2018

Thanks @mamedin for restarting dspace server.

@sevein -
Results from QA/testing:

@currmie currmie reopened this Oct 19, 2018

@currmie

This comment has been minimized.

Copy link

commented Oct 20, 2018

Hi @sevein,

  1. I just re-updated my installation again, re-created the DSpace space, and added a location for AIP storage.
  2. Got the expected wget output from http://dspace5x.archivematica.org:8080/xmlui/password-login.
  3. Ran a new transfer.
  4. Checked Git log for the latest commit deployed (5bd38f4) but found nothing.

For some reason, I'm not getting that latest commit/bug fix when I update. Not sure why that is happening. If that was resolved, then I think we could put this bugger to bed.

@sevein

This comment has been minimized.

Copy link
Contributor

commented Oct 20, 2018

If you're using compose:

$ cd src/archivematica && git pull --rebase
$ cd src/archivematica-storage-service && git pull --rebase

Then:

$ docker-compose up -d --force-recreate --build
$ make bootstrap
$ make restart-am-services

So similar to https://github.com/artefactual-labs/am/tree/master/compose#upgrading-to-the-latest-version-of-archivematica but it does not assume that the submodules are up to date!

@currmie

This comment has been minimized.

Copy link

commented Oct 20, 2018

Verified :)

[x ] AIP deposit in DSpace
[x ] Error-free

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.