Problem: Consistent Ingest failure with media/video transfer #966
Comments
|
Via @ablwr a possible change to the logging config could be |
ross-spencer
added
the
Type: bug
|
@jrwdunham I managed to find time to take a look at this today. As much as anything, it was to inform myself of the mechanics of the script and mediaconch for when I talk to the client. (Plus I really wanted to know what special features I'd find in 7mb of logging!!!) I took a couple of paths, but believe that I can confirm that this is a mediaconch error, not one of our own, and it has three parts:
To understand the mechanics of the FPR, I pulled the transcode script from the server and the validation script. I ran the transcode script to see if the new I then ran the validation script on the 16.12 (7423633 bytes) 17.07 (144707 bytes) The slightly intended bonus of removing the additional logging from the return here, is that the job doesn't kill SQL server: I don't have a solution as such, but I hope this helps with the conversation. Understanding this will help me chat to the client. Following your lead on the overall solution I think will still be best here. There might also be a consideration about database limits. I did a shallow amount of digging for maximum insert length and thing that even though it's buggy, the 7mb should not have send Percona away on its travels. I looked up this value |
|
Yeah, the MC version in use for AM is a good bit behind what is current/stable and I beliiiiiieve there was a notable refactoring from our initial slow data-parsing with XSL and it's now done directly in the C++ code. |
|
If you're already investing the time, it is probably worth doing a version bump for MediaConch to the latest, which is overall better and resolves plenty of other bugs (although that one bug we found a while back with the ambiguous SAR/Dar declaration is still in master but not latest stable release). |
Not yet (no sponsor for speed improvement so no ETA for the moment) As I understand, there is no more issues with test of MediaConch 17.07? FYI, a new release is planned next week, including the SAR declaration issue and false positive with FFV1 having Golomb Rice coder. |
|
Thanks Jerome! #goals Even if not done in C++ it is a lot more stable than before, there were several notable changes between current release and what AM is on. Archivematica 1.7 is also very imminent as I understand so that might have to wait until the next go-around. |
|
My sense is that adding the |
|
If it's solved, that's the best option, yes!!! |
|
@ablwr @jrwdunham I think we may still see a validation error with 16.12, Also, both, I tried to upgrade the package (probably very naively) inside the Docker container for the MCP-Client - It seems there maybe dependencies that can't be resolved in Ubuntu 14.04. IDK, do you know if it would work, say, if we updated the Dockerfile first? |
|
Without looking into it, I am pretty sure that is another issue similar to the FFV1 DAR_SAR failure -- an example of the specification itself changing, and MediaConch having to change validation policies to keep up with the development of the spec, hence not seeing it in 17.07. |
|
But EBML, MKV, and FFV1 standardization are much much closer to completion/finalization, I don't think there are any significant chances in any of them (esp. EBML/MKV), and MediaConch is in alignment with that (as we have had to keep up/alongside for the past 3 years!). An update would be good at this juncture, my opinion. And a bump to the forthcoming MediaConch release from 17.07 (and all other future releases) will be a lot less major than the bump between 16.12 and 17.07. |
|
@ablwr @ross-spencer I just ran a transfer through AM qa/1.x using as input a directory containing the lunchroom_manners_512kb.mp4 file in the archive that Ross links to in the original comment in this issue. Even when I change MediaConch 16.12 to use @ablwr if you want to continue investigating this, I'm happy to let you run with it. I have other things I can work on. I also have a PR that I'll post soon that changes the verbosity level to 4 because I think that's just a good idea, i.e., it fixes #807 and https://projects.artefactual.com/issues/11460. If it ends up that we need to do work on changing which version of MediaConch is installed (e.g., via osdeps, ansible and/or docker), I can help with that if need be. |
|
😎 Excellent! I'll give it a go although can't promise my machine is ready for (developer) primetime. |
|
I also tried to (also probably very naively) update MediaConch within Docker to see what an upgrade would look like, but, like @ross-spencer, got some Ubuntu failures for building the -dashboard component. I don't think it's related to this bug but it is similar in nature. I haven't yet looked at mediainfolib and an order of operations issue. Something else to consider in the existing mediaconch script is the use of the Another +1 for upgrading is better UTF-8 encoding handling, like in this bug fix: https://github.com/MediaArea/MediaInfoLib/pull/756/files I still advise upgrading to 17.12 (or if timing works out, 18.03 which will appear later this week). My only apprehension is because I'm not yet able to test it out myself and see how an upgrade would fare for the system in general. AFAIK (or maybe I should say as far as I remember) there was a breaking change a bit before 16.12 in how policies were delivered, but there haven't been any API changes since then, so upgrade should not be a problem. Anyway I think Archivematica uses the policies found within MediaConch and hasn't built custom policies on top, so it should not be a problem. |
Note: the feature in mediaconch-verbosity.feature implicitly requires a directory in archivematica-sampledata/ called mc-verbosity-test/ that contains a single .mp4 file called lunchroom_manners_512kb.mp4 which, when normalized to .mkv, will produce a file that makes MediaConch 16.12 but not 17.12 hang when performing an implementation check. The archive containing this file can be found at artefactual/archivematica#966. If possible, the AM-sampledata repo should be modified to include mc-verbosity-test/.
Adds a feature that demonstrates that Archivematica with MediaConch v. 17.12 installed can correctly validate (and document the validation of) a .mkv derivative that breaks MediaConch v. 16.12. Note: this feature implicitly requires a directory in archivematica-sampledata/ called mc-challenging-file/ that contains a single .mp4 file called lunchroom_manners_512kb.mp4 which, when normalized to .mkv, will produce a file that makes MediaConch 16.12 but not 17.12 hang when performing an implementation check. The archive containing this file can be found at artefactual/archivematica#966. If possible, the AM-sampledata repo should be modified to include mc-challenging-file/.
Adds a feature that demonstrates that Archivematica with MediaConch v. 17.12 installed can correctly validate (and document the validation of) a .mkv derivative that breaks MediaConch v. 16.12. Note: this feature implicitly requires a directory in archivematica-sampledata/ called mc-challenging-file/ that contains a single .mp4 file called lunchroom_manners_512kb.mp4 which, when normalized to .mkv, will produce a file that makes MediaConch 16.12 but not 17.12 hang when performing an implementation check. The archive containing this file can be found at artefactual/archivematica#966. If possible, the AM-sampledata repo should be modified to include mc-challenging-file/.
|
To QA:
|
We just (yesterday) released v18.03, which contains bug fixes for FFV1 (false positive errors). |
|
Thanks @JeromeMartinez, we've just filed #1006. |
|
Working! Tested with an ansible install which runs MediaConch 18.03 |
This package of data here is causing Ingest to fail in Archivematica 1.7:
To recreate:
If we look at the MCP Server Log we see a large chunk of MediaConch output, followed by a SQL failure:
This has been seen with other transfer material, but wasn't measured and recreated under the same control circumstances as here.
Will investigate more as I get an opportunity.
The text was updated successfully, but these errors were encountered: