Problem: newer pipelines may not be able to handle AIPs made with older pipelines #24

jrwdunham · 2018-07-20T23:28:24Z

Please describe the problem you'd like to be solved.

Sometimes an institution has been creating AIPs for a long time using various versions of Archivematica and the Storage Service and the institution wants to make sure that those AIPs are usable by their modern (currently installed) Archivematica version, e.g., in order to perform additional preservation actions like adding metadata.

Without modifying the content of such older AIPs, they may need to be compressed and given pointer files that document that compression.

Old AIPs may also have METS files that lack namespaces in their XML elements. It is unclear whether a modern Archivematica pipeline knows how to read such AIPs. Archivematica's METS interpretation functionality (in particular mets-reader-writer) may need to be modified in order to handle such AIPs.

Old AIPs may have accidentally been deleted from a Storage Service database. Their database tables in the SS db may need to be reconstructed from their METS files.

Similarly, old AIPs may have been created on a pipeline that has since been destroyed. Such AIPs may need their SS and MCP (pipeline) database tables reconstructed.

Describe the solution you'd like to see implemented.

I would like to see a case study of the techniques used (or attempted) to make Archivematica flexible and accommodating to various types of vintage AIP. I expect AIP re-ingest may be useful in compressing and generating pointer files for older AIPs, but API calls, Django management commands, and bespoke import/modernize scripts may all be necessary. I expect to see examples of using the AM and SS GUIs and APIs to interact with older AIPs and to discover whether older AIPs can be re-ingested.

Additional context

Imagine you had this:

Compressed?	Namespaced METS?	Typical vintage
0	1	AM >=1.2 (circa 2017 - 2018)
1	0	AM < 1.0 (2012-2013) / AM 1.0 - 1.1 (ca. 2014)
1	1	AM >= 1.2 (ca. 2015 - 2016)

How would you get to this:?

Compressed?	Pointer file?	Namespaced METS?	Typical vintage
1	1	1	AM = 1.7 (2018)

jrwdunham · 2018-07-31T23:25:52Z

Progress Update

@jhsimpson @sromkey

Three development branches in three different repos have been created to deal with this issue:

The most significant of these is the SS one, which introduces the import_aip Django management command which can import an existing (vintage) AIP into a running Storage Service instance, resulting in a compressed AIP with a pointer file. See import_aip

The general methodology employed here was to use import_aip to import exemplars of the various types of vintage AIP (i.e., types 0.0.1, 1.0.0, and 1.0.1) and then perform various actions and tests—in particular AIP re-ingest—to test whether Archivematica is able to handle these AIPs correctly.

Experiment 1: `0.0.1`: uncompressed, no pointer file, METS namespaced

exemplar AIP: new_uncompressed.tar.gz

import_aip command in a Docker-Compose environment:

$ make manage-ss ARG='import_aip /home/archivematica/new_uncompressed.tar.gz --decompress-source --compression-algorithm="7z with bzip"'

result: AIPs of this type appear to be functional in Archivematica after importation. Evidence:
- Unpackaging the imported AIP and diffing the resulting directory with the original directory shows they are identical:
```
$ diff -rq CVA1426-0fdedcc1-599d-4f6e-b132-4b7bc0355b1f/ tmp/CVA1426-0fdedcc1-599d-4f6e-b132-4b7bc0355b1f
```
- Initiating a full re-ingest on the imported AIP from the Storage Service succeeds.

Experiment 2: `1.0.0`: compressed, no pointer file, METS not namespaced

exemplar AIP: AM68-bdcb560d-7ddd-4c13-8040-1e565b4eddff.7z

import_aip command in a Docker-Compose environment:

$ make manage-ss ARG='import_aip /home/archivematica/AM68-bdcb560d-7ddd-4c13-8040-1e565b4eddff.7z --decompress-source --compression-algorithm="7z with bzip"'

result: AIPs of this type are not functional in Archivematica after importation. This appears to be due to the older METS files which, in addition to lacking namespaces, have other differences that make Archivematica choke. Evidence:

Attempts to perform a metadata re-ingest on the imported AIP from the Storage Service have failed at the micro-service Generate METS.xml document. Issues discovered:

Issue 1: The METS file of AIP AM68-bdcb560d-7ddd-4c13-8040-1e565b4eddff lacks a <mets:metsHdr> element. metsrw at branch dev/issue-24-handle-old-aips was modified to be able to handle this.
Issue 2: The METS file of AM68-bdcb560d-7ddd-4c13-8040-1e565b4eddff has a strange LABEL value in its <structMap>. Notice that the UUID of the
AIP (bdcb560d-7ddd-4c13-8040-1e565b4eddff) is repeated:
```
<div TYPE="directory" LABEL="AM68-bdcb560d-7ddd-4c13-8040-1e565b4eddff-bdcb560d-7ddd-4c13-8040-1e565b4eddff">
```

Issue 3: The METS file of AM68-bdcb560d-7ddd-4c13-8040-1e565b4eddff lacks <mets:div> elements for files—it only has <mets:fptr> elements. Compare this namespace-less <mets:structMap> to the following namespaced one:

<structMap TYPE="physical">
  <div TYPE="directory" LABEL="AM68-bdcb560d-7ddd-4c13-8040-1e565b4eddff-bdcb560d-7ddd-4c13-8040-1e565b4eddff">
    <div TYPE="directory" LABEL="objects">
      <fptr FILEID="V00154.MPG-3a6a182a-40a0-4c2b-9752-fc7e91ac1edf"/>
      <fptr FILEID="V00158.MPG-431913ba-4379-4373-8798-cc5f2b9dd769"/>
      <fptr FILEID="AM68.csv-fc0e52ca-a688-41c0-a10b-c1d36e21e804"/>
    </div>
  </div>
</structMap>

<mets:structMap ID="structMap_1" LABEL="Archivematica default" TYPE="physical">
  <mets:div LABEL="no-metadata-4060ee97-9c3f-4822-afaf-ebdf838284c3" TYPE="Directory">
    <mets:div LABEL="objects" TYPE="Directory">
      <mets:div LABEL="evelyn_s_photo-6383b731-99e0-432d-a911-a0d2dfd1ce76.tif" TYPE="Item">
        <mets:fptr FILEID="file-8140ebe5-295c-490b-a34a-83955b7c844e"/>
      </mets:div>
      <mets:div LABEL="evelyn_s_photo.jpg" TYPE="Item">
        <mets:fptr FILEID="file-ae8d4290-fe52-4954-b72a-0f591bee2e2f"/>
      </mets:div>
      <mets:div LABEL="metadata" TYPE="Directory">
        <mets:div LABEL="transfers" TYPE="Directory">
          <mets:div LABEL="no-metadata-46260807-ece1-4a0e-b70a-9814c701146b" TYPE="Directory"/>
        </mets:div>
      </mets:div>
      <mets:div LABEL="submissionDocumentation" TYPE="Directory">
        <mets:div LABEL="transfer-no-metadata-46260807-ece1-4a0e-b70a-9814c701146b" TYPE="Directory">
          <mets:div LABEL="METS.xml" TYPE="Item">
            <mets:fptr FILEID="file-590bd882-7521-498c-8f89-0958218f779d"/>
          </mets:div>
        </mets:div>
      </mets:div>
    </mets:div>
  </mets:div>
</mets:structMap>

The result of this is that clientScripts/archivematicaCreateMETSReingest.py will fail to loop through all of the files in the namespaceless METS file because mets.all_files() does not see any files. Cf. https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETSReingest.py#L25

TODO Next: continue to modify mets-reader-writer and AM::clientScripts/archivematicaCreateMETSReingest so that they can handle these older AIPs, using MCPClient/tests/test_reingest_mets_no_namespaces.py to confirm correct behaviour.

Experiment 3: `1.0.1`: compressed, no pointer file, METS namespaced

exemplar AIP: CVA180-12a-fa51e296-cb8f-4cfb-8260-3c8381ee4d32.7z

import_aip command in a Docker-Compose environment:

$ make manage-ss ARG='import_aip /home/archivematica/CVA180-12a-fa51e296-cb8f-4cfb-8260-3c8381ee4d32.7z --decompress-source --compression-algorithm="7z with bzip"'

result: AIPs of this type appear to be functional in Archivematica after importation. Evidence:
- Unpackaging the imported AIP and diffing the resulting directory with the original directory shows they are identical:
```
$ diff -rq CVA180-12a-fa51e296-cb8f-4cfb-8260-3c8381ee4d32 CVA180-12a-fa51e296-cb8f-4cfb-8260-3c8381ee4d32-2
```
- Initiating a full re-ingest on the imported AIP from the Storage Service succeeds.

sevein · 2018-12-03T23:16:29Z

@hakamine, I've updated JD's branches with a number of fixes. The main changes to support old METS:

METSRW: files are now recognized when <mets:fptr> lacks a <mets:div> wrapper
METSRW: added PREMIS v2.1 support

sevein · 2018-12-08T17:06:00Z

This is the latest re-ingested METS (using AM68's AIP): METS.xml.

There is a known issue with the validator:

[...]
ERROR ON LINE 2457: Element '{info:lc/xmlns/premis-v2}agent', attribute 'version': [facet 'enumeration'] The value '2.2' is not an element of the set {'2.0', '2.1'}.
ERROR ON LINE 2457: Element '{info:lc/xmlns/premis-v2}agent', attribute 'version': '2.2' is not a valid value of the atomic type '{info:lc/xmlns/premis-v2}versionSimpleType'.
ERROR ON LINE 2471: Element '{info:lc/xmlns/premis-v2}agent', attribute 'version': [facet 'enumeration'] The value '2.2' is not an element of the set {'2.0', '2.1'}.
ERROR ON LINE 2471: Element '{info:lc/xmlns/premis-v2}agent', attribute 'version': '2.2' is not a valid value of the atomic type '{info:lc/xmlns/premis-v2}versionSimpleType'.

I haven't found a way to combine in the same document elements using premis-v2-1.xsd and premis-v2-2.xsd. The validator seems to associate the namespace (info:lc/xmlns/premis-v2) only to the schema location appearing first.

CC @hakamine @evelynPM

hakamine · 2019-01-04T00:12:59Z

Hi @sevein I have a question re the error output shown in the previous comment. Do the errors show in the output of the import_aip command or the output of a different tool? I tried the import command on the same AIP and got a "successfully imported" message (no error messages). I am using: AM: dev/issue-24-handle-old-aips 20e2977 and SS: dev/issue-24-handle-old-aips f3fd8dd

sevein · 2019-01-04T15:25:04Z

Hi @hakamine! It's good news that it's working for you. The error mentioned above is only seen during the validation of the METS document generated by reingest of the imported AIP.

The validation issue has not been solved yet, unfortunately. We're still trying to find the best way to go. The root cause seems to be that you can only associate a XML prefix to a namespace once. E.g. if premis is used once and it points to premis-v2-1.xsd then that schema is used by the validator at all times regardless the fact that we are declaring different namespaces later on for the same prefix. So when we combine premis v2.1 + v2.2 (or v2.1 + v3.x as #370 suggests), the validator errors out. I've thought maybe we could use different prefixes, e.g. premis and premis3, but that may not be a good idea.

hakamine · 2019-01-07T20:15:46Z

When trying to index the AM68 AIP, getting the following error:

...
Rebuilding AIP UUID bdcb560d-7ddd-4c13-8040-1e565b4eddff
Processing AIP bdcb560d-7ddd-4c13-8040-1e565b4eddff
Deleting AIP bdcb560d-7ddd-4c13-8040-1e565b4eddff from aips/aip and aips/aipfile.
Command to run: ['unar', '-force-overwrite', '-o', '/tmp/tmpGzBMkl', '/mnt/siptransfer01-cvan110/aip
store-test-g543/bdcb/560d/7ddd/4c13/8040/1e56/5b4e/ddff/AM68-bdcb560d-7ddd-4c13-8040-1e565b4eddff.7z
', 'AM68-bdcb560d-7ddd-4c13-8040-1e565b4eddff/data/METS.bdcb560d-7ddd-4c13-8040-1e565b4eddff.xml']
/mnt/siptransfer01-cvan110/aipstore-test-g543/bdcb/560d/7ddd/4c13/8040/1e56/5b4e/ddff/AM68-bdcb560d-
7ddd-4c13-8040-1e565b4eddff.7z: 7-Zip
  AM68-bdcb560d-7ddd-4c13-8040-1e565b4eddff/data/METS.bdcb560d-7ddd-4c13-8040-1e565b4eddff.xml  (715
76 B)... OK.
Successfully extracted to "/tmp/tmpGzBMkl/AM68-bdcb560d-7ddd-4c13-8040-1e565b4eddff".
Removed FITS output from METS.
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packag
es/django/core/management/__init__.py", line 354, in execute_from_command_line
    utility.execute()
  File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packag
es/django/core/management/__init__.py", line 346, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packag
es/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packag
es/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/opt/archivematica/archivematica/src/dashboard/src/main/management/commands/rebuild_elastics
earch_aip_index_from_files.py", line 295, in handle
    delete_existing_data=options['delete'],
  File "/opt/archivematica/archivematica/src/dashboard/src/main/management/commands/rebuild_elastics
earch_aip_index_from_files.py", line 169, in processAIPThenDeleteMETSFile
    size=aip_info[0]['size'],
  File "/usr/lib/archivematica/archivematicaCommon/elasticSearchFunctions.py", line 425, in index_ai
p
    mets_created_attr = mets_hdr.get('CREATEDATE')
AttributeError: 'NoneType' object has no attribute 'get'

It seems that the script is looking for an attribute that is not present in the METS file

sevein · 2019-01-07T21:05:51Z

@hakamine thanks for the report. I've just added a new commit to handle that scenario where metsHdr is missing. See artefactual/archivematica@ce1579f.

hakamine · 2019-01-07T23:28:28Z

Thank you @sevein for the fix. I tested it and the indexing of management the AIP "AM68" (using the custom management command rebuild_elasticsearch_aip_index_from_files) finished without errors this time. However, I am getting errors when trying to download individual files from the AIP from the archival storage tab:

When I try to download any of the files, getting an internal server error, the dashboard log shows a message like:

ERROR     2019-01-07 23:11:02  django.request:base:handle_uncaught_exception:256:  Internal Server Error: /archival-storage/download/aip/file/431913ba-4379-4373-8798-cc5f2b9dd769/
Traceback (most recent call last):
 File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response
   response = wrapped_callback(request, *callback_args, **callback_kwargs)
 File "/opt/archivematica/archivematica/src/dashboard/src/components/archival_storage/views.py", line 318, in aip_file_download
   file = models.File.objects.get(uuid=uuid)
 File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
   return getattr(self.get_queryset(), name)(*args, **kwargs)
 File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packages/django/db/models/query.py", line 334, in get
   self.model._meta.object_name
DoesNotExist: File matching query does not exist.

(not sure if this could be caused by the "vintage" METS or by a bug in the reindex command script that affects all AIPs. If you think it could be the latter let me know and I'll open a separate github issue)

sevein · 2019-01-07T23:36:22Z

Thanks, I'll take a look!

sevein · 2019-01-18T22:06:51Z

Progress update!

The PREMIS3 work is tentatively complete - further testing may reveal more issues.
New METS after re-ingest is now valid, compare original and after re-ingest documents.

Known issues:

Re-ingest may result in invalid bags - this is a problem in Storage Service and/or python-bagit, Problem: bagit created by Storage Service after reingest may have outdated tag manifest files #449.
metsHdr uses CREATEDATE instead of LASTMODDATE
@hakamine's investigating some indexing issues.

hakamine · 2019-01-23T22:53:22Z

@sevein would it be ok if I rebase AM dev/issue-24-handle-old-aips on top of current qa/1.x ? I would like to check if the ES upgrade changes fix the problem with the indexing of AIP files.

sevein · 2019-01-23T22:55:34Z

Please do! Thanks.

hakamine · 2019-02-11T18:55:54Z

It looks like there is something in AM dev/issue-24-handle-old-aips break indexing of AIP files. I did the following test for both AM qa/1.x (c6396b9) and dev/issue-24-handle-old-aips (d4ac060) branches:

Delete the aips and aipfiles indexes
Run the rebuild_elasticsearch_aip_index_from_files management command for an ingested AIP containing the "Images" sampledata directory

While the management command finishes without errors for both branches, the resulting aipfiles index in branch AM dev/issue-24-handle-old-aips has all the FILEUUID fields with value null. This causes an internal server error when the dashboard's Archival Storage tab when "Show files" is checked, dashboard log is:

ERROR     2019-02-09 00:20:15  django.request:base:handle_uncaught_exception:256:  Internal Server Error: /archival-storage/search/
Traceback (most recent call last):
  File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/archivematica/archivematica/src/dashboard/src/components/archival_storage/views.py", line 180, in search
    'page': page_data,
  File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packages/django/shortcuts.py", line 67, in render
    template_name, context, request=request, using=using)
...
 File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packages/django/template/defaulttags.py",
line 493, in render
    url = reverse(view_name, args=args, kwargs=kwargs, current_app=current_app)
  File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packages/django/core/urlresolvers.py", lin
e 578, in reverse
    return force_text(iri_to_uri(resolver._reverse_with_prefix(view, prefix, *args, **kwargs)))
  File "/usr/share/archivematica/virtualenvs/archivematica-dashboard/local/lib/python2.7/site-packages/django/core/urlresolvers.py", lin
e 495, in _reverse_with_prefix
    (lookup_view_s, args, kwargs, len(patterns), patterns))
NoReverseMatch: Reverse for 'components.archival_storage.views.aip_file_download' with arguments '(None,)' and keyword arguments '{}' no
t found. 1 pattern(s) tried: ['archival-storage/download/aip/file/(?P<uuid>[\\w]{8}(-[\\w]{4}){3}-[\\w]{12})/$']

aipfiles index is attached (dump produced with, e.g.,curl -o search_aipfiles_reindex_devissue24.json http://localhost:9200/aipfiles/_search?pretty=true&q=*:* )(for comparison, the aipfiles index obtained with branch qa/1.x is also attached.
search_aipfiles_reindex_devissue24.json.zip
search_aipfiles_reindex_qa1x.json.zip

hakamine · 2019-02-15T18:59:39Z

@sevein I added some commits to AM branch dev/issue-24-aip-index-premis2-fallback (which is based off dev/index-24-handle-old-aips), in order to try to fix the indexing issues mentioned in the previous comment. So far it seems to be working (I'll continue testing with other AIPs). Please let me know if the code looks good and I'll merge these commits to branch dev/index-24-handle-old-aips

It looks we are getting closer to the goal! Should we create PRs for the AM/SS/metsrw dev/issue-24-handle-old-aips branches?

Update 1: while the index worked for the "images" sampledata, I am having some problems with AIP AM68 files.

Update 2: the problem with the AM68 AIP is detailed in #504

Update 3: added a commit to dev/issue-24-aip-index-premis2-fallback that fixes #504

Update 4: tested the AIP import and reindex management commands with a few more AIPs as shown in the screenshot below. Both the import and reindex commands completed without error messages for all of these tested. However, there is an issue with the "Date stored" value in some AIPs, instead of the date they were originally ingested, it shows the date of the reindex (red box in the screenshot)

This occurs for AIPs in group G5 (AIPs) (ingested between 2011/11-2012/09, no namespaces in the METS file, AIPs "AM68" "CVA4-6-7" "AM1553-MI-515" in the screenshot ). However AIPs in group G4 (ingested in 2014, no namespaces in the METS file, AIP "1970-03-03" "1979-09-13_SPEC" in the screenshot) show the correct value for the date.
Taking a look at the METS files, in the G5 AIPs the metsHdr CREATEDATE is not present (i.e., the AM version used to ingest these AIPs didn't include it in the METS file), so the reindex script just uses the current date (the date the reindex command runs) in the aips index created field. In conclusion, this is not a bug of the reindex script (it's a limitation caused by lack of information in the METS files)

sevein · 2019-02-20T20:13:30Z

@hakamine nice job on dev/issue-24-aip-index-premis2-fallback, thanks for your comment and all the updates. It's all working nicely locally so I'd suggest to update dev/index-24-handle-old-aips with all your changes.

sevein · 2019-02-20T20:20:42Z

@hakamine, FYI Evelyn noticed that after reingest we have metsHdr created as follows:

<mets:metsHdr CREATEDATE="2019-02-20T19:57:40"/>

It may be preferably to use LASTMODDATE instead (see https://github.com/artefactual-labs/mets-reader-writer/blob/1f4b7fdbb37512f2c7c42a750b9f08bc73738d9f/metsrw/mets.py#L183-L192), but we won't be addressing that now since it's not a deal breaker.

hakamine · 2019-02-21T18:32:38Z

Thank you @sevein, dev/issue-24-handle-old-aips has been updated (and deleted dev/issue-24-aip-index-premis-fallback)

sevein · 2019-03-10T19:34:52Z

Update:

metsrw Handle old Archivematica METS documents artefactual-labs/mets-reader-writer#53 (fixes released @ v0.3.0)
am: Use PREMIS 3 and fixes for old AIPs artefactual/archivematica#1365 (merged to qa/1.x)
ss: Add import_aip management command artefactual/archivematica-storage-service#396 (merged to qa/0.x)

sallain · 2019-07-08T21:01:58Z

@hakamine can you take a look at this and change the label to Done if you feel confident that this is resolved?

hakamine · 2019-07-12T21:32:30Z

Tested on AM qa/1.x 50affa (1.10.0-rc.1) SS qa/0.x fbdf31 (0.15.0-rc.1). Import and reindex of CVA old AIPs is working.

jrwdunham self-assigned this Jul 20, 2018

jrwdunham added Type: feature New functionality. Status: refining The issue needs additional details to ensure that requirements are clear. labels Jul 20, 2018

jrwdunham changed the title ~~Problem: it is difficult to "modernize" OLD AIPs~~ Problem: AIPs made with older pipelines are not compatible with newer pipelines Jul 20, 2018

jrwdunham changed the title ~~Problem: AIPs made with older pipelines are not compatible with newer pipelines~~ Problem: newer pipelines may not be able to handle AIPs made with older pipelines Jul 21, 2018

jrwdunham mentioned this issue Jul 25, 2018

Add import_aip management command artefactual/archivematica-storage-service#396

Merged

jhsimpson added Status: in progress Issue that is currently being worked on. and removed Status: refining The issue needs additional details to ensure that requirements are clear. labels Jul 25, 2018

jrwdunham added the CVA label Jul 25, 2018

sromkey mentioned this issue Aug 1, 2018

Problem: Reingest assumes standard transfer type #51

Open

ablwr mentioned this issue Aug 13, 2018

Feature: Add new api endpoint for AIP Migration #71

Closed

sromkey added Status: blocked Work on the issue is not possible until the blocking condition is removed. and removed Status: in progress Issue that is currently being worked on. labels Aug 23, 2018

sevein mentioned this issue Dec 3, 2018

Handle old Archivematica METS documents artefactual-labs/mets-reader-writer#53

Merged

sevein added Status: in progress Issue that is currently being worked on. and removed Status: blocked Work on the issue is not possible until the blocking condition is removed. labels Jan 18, 2019

sevein mentioned this issue Jan 18, 2019

AIP METS file has 2 different versions of premis #370

Closed

sevein mentioned this issue Jan 22, 2019

Problem: bagit created by Storage Service after reingest may have outdated tag manifest files #449

Closed

hakamine mentioned this issue Feb 15, 2019

Problem: can't download AIP files from Archival Storage tab link if the AIP file isn't listed in the MCP Files table #504

Closed

sromkey added the triage-release-1.10 label Feb 19, 2019

sevein assigned sevein and hakamine and unassigned jrwdunham Feb 20, 2019

sevein mentioned this issue Mar 10, 2019

Use PREMIS 3 and fixes for old AIPs artefactual/archivematica#1365

Merged

sevein removed the triage-release-1.10 label Mar 12, 2019

sevein added this to the 1.10.0 milestone Mar 12, 2019

hakamine mentioned this issue Mar 26, 2019

Problem: AIP index error: ''Limit of total fields [1000] in index [aips] has been exceeded'' #608

Closed

hakamine added Status: review The issue's code has been merged and is ready for testing/review. and removed Status: in progress Issue that is currently being worked on. labels Mar 29, 2019

This was referenced Apr 23, 2019

Fix aips and aipfiles ES mappings artefactual/archivematica#1405

Closed

Investigate possible ES mapping changes in qa/1.x branch #619

Closed

sevein removed their assignment May 10, 2019

hakamine removed the Status: review The issue's code has been merged and is ready for testing/review. label Jul 12, 2019

hakamine closed this as completed Jul 12, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problem: newer pipelines may not be able to handle AIPs made with older pipelines #24

Problem: newer pipelines may not be able to handle AIPs made with older pipelines #24

jrwdunham commented Jul 20, 2018 •

edited

Loading

jrwdunham commented Jul 31, 2018 •

edited

Loading

sevein commented Dec 3, 2018 •

edited

Loading

sevein commented Dec 8, 2018 •

edited

Loading

hakamine commented Jan 4, 2019

sevein commented Jan 4, 2019 •

edited

Loading

hakamine commented Jan 7, 2019

sevein commented Jan 7, 2019

hakamine commented Jan 7, 2019

sevein commented Jan 7, 2019

sevein commented Jan 18, 2019 •

edited

Loading

hakamine commented Jan 23, 2019

sevein commented Jan 23, 2019

hakamine commented Feb 11, 2019

hakamine commented Feb 15, 2019 •

edited

Loading

sevein commented Feb 20, 2019

sevein commented Feb 20, 2019

hakamine commented Feb 21, 2019

sevein commented Mar 10, 2019 •

edited by hakamine

Loading

sallain commented Jul 8, 2019

hakamine commented Jul 12, 2019

Problem: newer pipelines may not be able to handle AIPs made with older pipelines #24

Problem: newer pipelines may not be able to handle AIPs made with older pipelines #24

Comments

jrwdunham commented Jul 20, 2018 • edited Loading

jrwdunham commented Jul 31, 2018 • edited Loading

Progress Update

Experiment 1: 0.0.1: uncompressed, no pointer file, METS namespaced

Experiment 2: 1.0.0: compressed, no pointer file, METS not namespaced

Experiment 3: 1.0.1: compressed, no pointer file, METS namespaced

sevein commented Dec 3, 2018 • edited Loading

sevein commented Dec 8, 2018 • edited Loading

hakamine commented Jan 4, 2019

sevein commented Jan 4, 2019 • edited Loading

hakamine commented Jan 7, 2019

sevein commented Jan 7, 2019

hakamine commented Jan 7, 2019

sevein commented Jan 7, 2019

sevein commented Jan 18, 2019 • edited Loading

hakamine commented Jan 23, 2019

sevein commented Jan 23, 2019

hakamine commented Feb 11, 2019

hakamine commented Feb 15, 2019 • edited Loading

sevein commented Feb 20, 2019

sevein commented Feb 20, 2019

hakamine commented Feb 21, 2019

sevein commented Mar 10, 2019 • edited by hakamine Loading

sallain commented Jul 8, 2019

hakamine commented Jul 12, 2019

jrwdunham commented Jul 20, 2018 •

edited

Loading

jrwdunham commented Jul 31, 2018 •

edited

Loading

Experiment 1: `0.0.1`: uncompressed, no pointer file, METS namespaced

Experiment 2: `1.0.0`: compressed, no pointer file, METS not namespaced

Experiment 3: `1.0.1`: compressed, no pointer file, METS namespaced

sevein commented Dec 3, 2018 •

edited

Loading

sevein commented Dec 8, 2018 •

edited

Loading

sevein commented Jan 4, 2019 •

edited

Loading

sevein commented Jan 18, 2019 •

edited

Loading

hakamine commented Feb 15, 2019 •

edited

Loading

sevein commented Mar 10, 2019 •

edited by hakamine

Loading