Problem: currentlyProcessing is not empty after transfer is done

[jorikvankemenade]

Expected behaviour
When a transfer has completed successfully all traces are removed from the currentlyProcessing directory.

Current behaviour
When a transfer has completed successfully a folder containing some of the by-products remain in the currentlyProcessing directory. An example:

[root@archivematica-afs ~]# tree  /volume/sharedDirectory/currentlyProcessing/Images-aee6018d-ba65-4291-8aab-231309876148/
/volume/sharedDirectory/currentlyProcessing/Images-aee6018d-ba65-4291-8aab-231309876148/
├── logs
│   ├── bulk-0b9aa156-6e03-468a-95db-44e307e79306
│   │   ├── domain_histogram.txt
│   │   ├── domain.txt
│   │   ├── report.xml
│   │   ├── url_histogram.txt
│   │   ├── url_services.txt
│   │   └── url.txt
│   ├── bulk-3df51291-6e2d-4f3e-8294-4ec3b5a74ced
│   │   └── report.xml
│   ├── bulk-59f01e88-9d00-4e94-b4ce-73516bd65b01
│   │   ├── domain_histogram.txt
│   │   ├── domain.txt
│   │   └── report.xml
│   ├── bulk-5da098b7-616d-4bfe-a375-7bc4f82f94f0
│   │   ├── report.xml
│   │   ├── telephone_histogram.txt
│   │   └── telephone.txt
│   ├── bulk-6071c87a-186d-4c09-b881-3fe25b6e4d9f
│   │   ├── domain_histogram.txt
│   │   ├── domain.txt
│   │   └── report.xml
│   ├── bulk-82ca71d6-bbc5-45d9-96ae-f1041e85f297
│   │   ├── exif.txt
│   │   └── report.xml
│   ├── bulk-ca32e7ca-28e8-4230-8097-25165eacf5ac
│   │   ├── exif.txt
│   │   ├── report.xml
│   │   ├── telephone_histogram.txt
│   │   └── telephone.txt
│   ├── bulk-cf6b996e-7ee4-4885-8525-6fe281040658
│   │   └── report.xml
│   ├── bulk-e0d4f869-e190-4169-8fe1-f7a852ce2820
│   │   ├── domain_histogram.txt
│   │   ├── domain.txt
│   │   ├── report.xml
│   │   ├── url_histogram.txt
│   │   ├── url_services.txt
│   │   └── url.txt
│   ├── bulk-e816a377-360a-481d-b717-f994e738f06f
│   │   ├── exif.txt
│   │   └── report.xml
│   ├── fileFormatIdentification.log
│   ├── fileMeta
│   └── filenameCleanup.log
├── metadata
│   ├── directory_tree.txt
│   └── submissionDocumentation
│       └── METS.xml
├── objects
└── processingMCP.xml

When comparing the workflow of a qa/1.x with a 10.1 version I see that there is a difference in the workflow. Is it possible that some step that should clean up this file has accidentally been removed from the workflow?

qa

10.1

Steps to reproduce
Run any sample transfer and check the contents of the currentlyProcessing directory.

Your environment (version of Archivematica, operating system, other relevant details)
Latest qa/1.x branch

---

For Artefactual use:

Before you close this issue, you must check off the following:

• All pull requests related to this issue are properly linked
• All pull requests related to this issue have been merged
• A testing plan for this issue has been implemented and passed (testing plan information should be included in the issue body or comments)
• Documentation regarding this issue has been written and merged (if applicable)
• Details about this issue have been added to the release notes (if applicable)

[sromkey]

@jorikvankemenade we've marked this Low severity because while we recognize it's an issue it's never really come up as a major problem or hit on performance- but can you let us know if you have found otherwise.

[jorikvankemenade]

@sromkey I understand the priority. With about 0.5MB per transfer, it is not a big problem. Even when running a lot of transfers it is manageable. I would like to suggest to mark it as a regression since it is introduced on the QA branch.

[jorikvankemenade]

I just did some digging. I am sure that the problem is the last "Move to SIP creation directory for completed transfers". This moves the folder structure I just described to sharedDirectory/watchedDirectories/SIPCreation/completedTransfers/. I think this step is lost in the queuing work that has been merged around September. So maybe we can fix this with the work we are doing for #1055 and #1108.

[jorikvankemenade]

The step I am talking about is removed in this PR. We might have to move this step back into the workflow, but I am not sure if this has any consequences. The PR description also briefly mentions this problem, but not really any solution to it.

Before fixing this, it might be good to think if the whole completed transfer folder is wanted/used. If this is not the case then we need to make sure that this "extra" folder is deleted, rather then moved.

[FransPicturae]

In our local environment we reverted the PR @jorikvankemenade mentioned, the behavior of not deleting the currentlyProccessing contents remains. This bug is preventing new transfers to be started in our environment. For us this is high impact.

[jorikvankemenade]

This bug is preventing new transfers to be started in our environment.

Could you maybe clarify this a bit? How is not deleting the currently processing contents preventing new transfers from being started?

Or is the problem of transfers not being started new, and caused by reverting the PR. If this is the case, I wouldn't be surprised. Reverting that PR changes the workflow. This can cause problems if this means that the workflow never hits a terminal link. See #1055 and this PR for some extra information on the termination of workflows.

[FransPicturae]

By reverting the PR we investigated if the currentlyProccesing directory would be empty after a succesful transfer/ingest. When a transfer is done the SIP is copied and gets a new uuid. The old remains in the folder. We discovered that manual removing the files would make it possible to start a new transfer. Which was not possible using some custom MicroServices.

[ross-spencer]

Comment on the forum about the correctness of the database at this stage in the transfer too. It'd be a good idea to peek into it a bit more with some deeper analysis: https://groups.google.com/g/archivematica/c/f3C_2gdgY0U/m/PUBZbF3dAAAJ

[arthytrip]

I confirm the same functioning observed by Jorik and FransPicturae.
I also confirm that in the database the packages (Transfers table) all remain with currentLocation set to "'%sharedPath%currentlyProcessing/...".
N.B. Please also support DB. It is not true that it is only transitory, for the current management and the related reporting it is essential!