media files not being removed from alps after batch ingest

[jenyoung]

Severity

Is the production site running?

• yes
• no

Are staff blocked from performing their work?

• yes
• no

Descriptive summary

After a successful batch ingest, media files are not being automatically removed from the folder on alps.

Expected behavior

Media files removed from folder upon successful batch ingest.

Actual behavior

Media files remain in alps folder after successful batch ingest

Folders with this issue:

• Dave Eanet
• Imagination without borders
• J Rothfuss

[davidschober]

@jenyoung is this occurring on staging anywhere?

[davidschober]

@mbklein and @Toputnal reopening this for investigation. It looks like the problem is still occurring @jyo ran into the problem again today.

[ccaizzi]

can we look into this early next week please?

[mbklein]

I'm looking.

[davidschober]

mbklein [11:32 AM]
@d-venckus The space issue on alps isn’t entirely masterfile content. There’s only 0.5TB in the masterfiles directory.

[11:33]
Matterhorn’s working directory (which is on the same partition) is at 2.4TB.

[11:33]
Almost all of which can be purged.

mbklein [11:39 AM]
There’s even a script to do it. https://wiki.dlib.indiana.edu/display/VarVideo/Maintaining+Matterhorn+Disk+Space

[11:41]
The defaults even apply to how and where everything is installed on Alps.

Per conversation above, I'll kick off a new issue to run the matterhorn clean out script.

[jenyoung]

my test: i tested batch ingesting one audio file. the ingest was successful but the audio file is still sitting in the Alps folder (jen_test). the file has not been removed. it was ingested on friday 12/16.

[ccaizzi]

@mbklein apart from the cleanup script, it appears master files are still remaining. will you continue to look into this today please.

[mbklein]

@d-venckus @Toputnal User avalon still does not have write access to /farchive/farchive20.

[avalon@avalonweb2p]$ touch /farchive/farchive20/numedia_41058-jy_test_audio.wav
touch: cannot touch `/farchive/farchive20/numedia_41058-jy_test_audio.wav': Permission denied

Also, when I try to ls -la /farchive/farchive20 as avalon, I get a bunch of Permission denied errors followed by a partial directory listing with borked permission masks:

total 0
d????????? ? ? ? ?            ? .
d????????? ? ? ? ?            ? ..
-????????? ? ? ? ?            ? 1009766_musica_01_01_01.wav
-????????? ? ? ? ?            ? 1009766_musica_01_01_02.wav
-????????? ? ? ? ?            ? 1009766_musica_01_01_03.wav
-????????? ? ? ? ?            ? 1009766_musica_01_01_04.wav
-????????? ? ? ? ?            ? 1009766_musica_01_01_05.wav
-????????? ? ? ? ?            ? 1009766_musica_01_01_06.wav
-????????? ? ? ? ?            ? 1009766_musica_01_01_07.wav
-????????? ? ? ? ?            ? 1009766_musica_01_02_01.wav
-????????? ? ? ? ?            ? 1009766_musica_01_02_02.wav
-????????? ? ? ? ?            ? 1009766_musica_01_02_03.wav
-????????? ? ? ? ?            ? 1009766_musica_01_02_04.wav
-????????? ? ? ? ?            ? 1009766_musica_01_02_05.wav
-????????? ? ? ? ?            ? 1009766_musica_01_02_06.wav
... (continues) ...

The permissions look normal when I ls as root, but I've never seen a situation where a user could list files but not stat them.

[Toputnal]

fixed @mbklein . T'was ye olde sticky bit which bit us!

[ccaizzi]

@mbklein should we test once more? @Nic4Images and @jenyoung want to try running another test?

[mbklein]

I'll do a manual move using the avalon account, and then we can try an automated one.

[mbklein]

OK, that's done. 😄 @Nic4Images @jenyoung feel free to try another.

[Nic4Images]

Jen and I are on it. Update to come after 11AM!

[jenyoung]

@mbklein tested a video and audio file batch. went in and were removed from the dropbox as expected.