Encoding leaving behind tons of extra directories in the HLS directory

[KyleMaas]

Describe the issue
Sometimes (and I haven't figured out what conditions cause this) the HLS directory will contain a whole bunch of additional directories of video in the format {media MD5 - 32 characters}{some seemingly-random alphanumeric string - 9 characters}. Here's an example of just one video that this happened with:

$ ls media_files/hls | grep b5b331f7f7494a29897a3
b5b331f7f7494a29897a3b04161857eb
b5b331f7f7494a29897a3b04161857eb0BPxEy8Ji
b5b331f7f7494a29897a3b04161857eb0eryN6SKU
b5b331f7f7494a29897a3b04161857eb0GpvNvAqa
b5b331f7f7494a29897a3b04161857eb0I568t0NT
b5b331f7f7494a29897a3b04161857eb0jOqfNH0w
b5b331f7f7494a29897a3b04161857eb0ms6PHOgn
b5b331f7f7494a29897a3b04161857eb1AxX2g43J
b5b331f7f7494a29897a3b04161857eb1Cu1o18cJ
b5b331f7f7494a29897a3b04161857eb2Y8zXoG4c
b5b331f7f7494a29897a3b04161857eb3Kfih5nBy
b5b331f7f7494a29897a3b04161857eb3RVfu3s2u
b5b331f7f7494a29897a3b04161857eb3sMjYw8TK
b5b331f7f7494a29897a3b04161857eb3v0YlN5ft
b5b331f7f7494a29897a3b04161857eb40yp7yVt0
b5b331f7f7494a29897a3b04161857eb505OdWCdw
b5b331f7f7494a29897a3b04161857eb5wR1rIqCA
b5b331f7f7494a29897a3b04161857eb5Y0FbzcGW
b5b331f7f7494a29897a3b04161857eb6F2Y0oj1D
b5b331f7f7494a29897a3b04161857eb6jneaGBOR
b5b331f7f7494a29897a3b04161857eb6VOTm5ujm
b5b331f7f7494a29897a3b04161857eb7gHsLVKlC
b5b331f7f7494a29897a3b04161857eb7HAdV8QVp
b5b331f7f7494a29897a3b04161857eb7w2QHGbNn
b5b331f7f7494a29897a3b04161857eb83xip1HD5
b5b331f7f7494a29897a3b04161857eb9914VriuM
b5b331f7f7494a29897a3b04161857eb9gYzso9YF
b5b331f7f7494a29897a3b04161857ebAjEI8tkh1
b5b331f7f7494a29897a3b04161857ebAl15xRj1y
b5b331f7f7494a29897a3b04161857ebARIm1ggNh
b5b331f7f7494a29897a3b04161857ebaYqf7UyPq
b5b331f7f7494a29897a3b04161857ebBkq13wCVB
b5b331f7f7494a29897a3b04161857ebbtHdcxJPk
b5b331f7f7494a29897a3b04161857ebBW2T312dL
b5b331f7f7494a29897a3b04161857ebc2su8CylJ
b5b331f7f7494a29897a3b04161857ebcMjbzZw9O
b5b331f7f7494a29897a3b04161857ebCNYIuzFEX
b5b331f7f7494a29897a3b04161857ebcziSYJRZN
b5b331f7f7494a29897a3b04161857ebD2g5Irn1K
b5b331f7f7494a29897a3b04161857ebD5UzrI5Ut
b5b331f7f7494a29897a3b04161857ebdjvTgnwyk
b5b331f7f7494a29897a3b04161857ebDpSfpKQdb
b5b331f7f7494a29897a3b04161857ebdY199r1rA
b5b331f7f7494a29897a3b04161857ebE5jyk4LIp
b5b331f7f7494a29897a3b04161857ebeDaAU5XDI
b5b331f7f7494a29897a3b04161857ebEfA5CgREK
b5b331f7f7494a29897a3b04161857ebEfo6XDeAn
b5b331f7f7494a29897a3b04161857ebeNa6kl6Bg
b5b331f7f7494a29897a3b04161857ebEP37ekj4m
b5b331f7f7494a29897a3b04161857ebewDOkmr0b
b5b331f7f7494a29897a3b04161857ebExfT7wl1N
b5b331f7f7494a29897a3b04161857ebf00bwPchg
b5b331f7f7494a29897a3b04161857ebF7MilwbqZ
b5b331f7f7494a29897a3b04161857ebFD7I0oQPC
b5b331f7f7494a29897a3b04161857ebfGyvERRFI
b5b331f7f7494a29897a3b04161857ebflHo0CfNC
b5b331f7f7494a29897a3b04161857ebFWyPNtfos
b5b331f7f7494a29897a3b04161857ebfXLYyHtl3
b5b331f7f7494a29897a3b04161857ebG1hgQIfH6
b5b331f7f7494a29897a3b04161857ebghUxPJcyf
b5b331f7f7494a29897a3b04161857ebGQoi5Lu6d
b5b331f7f7494a29897a3b04161857ebgQVvi7EV2
b5b331f7f7494a29897a3b04161857ebh3WJxsG5f
b5b331f7f7494a29897a3b04161857ebHBgZsFBVR
b5b331f7f7494a29897a3b04161857ebHBwdba6XB
b5b331f7f7494a29897a3b04161857ebHh9Njrya3
b5b331f7f7494a29897a3b04161857ebHIjL1Ib7S
b5b331f7f7494a29897a3b04161857ebhkJSLaGi1
b5b331f7f7494a29897a3b04161857ebHWUvrdHA9
b5b331f7f7494a29897a3b04161857ebHyBtum5js
b5b331f7f7494a29897a3b04161857ebI1h9rRYCK
b5b331f7f7494a29897a3b04161857ebIevbFNurU
b5b331f7f7494a29897a3b04161857ebIjQY9ctT6
b5b331f7f7494a29897a3b04161857ebJ3wql8ifr
b5b331f7f7494a29897a3b04161857ebJgOA2hF0L
b5b331f7f7494a29897a3b04161857ebJUcx3gLq6
b5b331f7f7494a29897a3b04161857ebJWB3lYIab
b5b331f7f7494a29897a3b04161857ebjxGlpefBm
b5b331f7f7494a29897a3b04161857ebLAF16ByNM
b5b331f7f7494a29897a3b04161857ebLtCLbC7bT
b5b331f7f7494a29897a3b04161857eblZ9MCvTbs
b5b331f7f7494a29897a3b04161857ebm4yYvT803
b5b331f7f7494a29897a3b04161857ebmGvcDrmKW
b5b331f7f7494a29897a3b04161857ebN4uajbf7d
b5b331f7f7494a29897a3b04161857ebn5kXMFccY
b5b331f7f7494a29897a3b04161857ebN9PXpps4L
b5b331f7f7494a29897a3b04161857ebnbqQb8zCQ
b5b331f7f7494a29897a3b04161857ebnEdZurDQl
b5b331f7f7494a29897a3b04161857ebniWFxky77
b5b331f7f7494a29897a3b04161857ebNOo8EEcnw
b5b331f7f7494a29897a3b04161857ebNyuHaU6Qa
b5b331f7f7494a29897a3b04161857eboeHQVZfV6
b5b331f7f7494a29897a3b04161857ebOmMwPa8kP
b5b331f7f7494a29897a3b04161857ebopyhPHRyM
b5b331f7f7494a29897a3b04161857ebOUso1gBov
b5b331f7f7494a29897a3b04161857ebOVLW3iYMB
b5b331f7f7494a29897a3b04161857eboXtIW8QDF
b5b331f7f7494a29897a3b04161857ebP7GbtuiPh
b5b331f7f7494a29897a3b04161857ebPUbSrM7La
b5b331f7f7494a29897a3b04161857ebPzyDxlDEs
b5b331f7f7494a29897a3b04161857ebqBiqvw4OH
b5b331f7f7494a29897a3b04161857ebQbqT8sWfQ
b5b331f7f7494a29897a3b04161857ebqdeceDr3Q
b5b331f7f7494a29897a3b04161857ebrag7VEajl
b5b331f7f7494a29897a3b04161857ebrEYoDfIO0
b5b331f7f7494a29897a3b04161857ebrFH4TqDYq
b5b331f7f7494a29897a3b04161857ebrItnhDEjT
b5b331f7f7494a29897a3b04161857ebrnzWdp2fd
b5b331f7f7494a29897a3b04161857ebrU6j1mJbL
b5b331f7f7494a29897a3b04161857ebRwToB3Bi6
b5b331f7f7494a29897a3b04161857ebS5qKaR4Lv
b5b331f7f7494a29897a3b04161857ebS9ucZWWeE
b5b331f7f7494a29897a3b04161857ebSDBJgl89Q
b5b331f7f7494a29897a3b04161857ebsg7ntHB2N
b5b331f7f7494a29897a3b04161857ebsJlRuI9Ec
b5b331f7f7494a29897a3b04161857ebskx0FDkyH
b5b331f7f7494a29897a3b04161857ebsLuOpRaUM
b5b331f7f7494a29897a3b04161857ebSmDOZrfob
b5b331f7f7494a29897a3b04161857ebSpATe5RcH
b5b331f7f7494a29897a3b04161857ebSpKB6F2Hn
b5b331f7f7494a29897a3b04161857ebStJzeG0RV
b5b331f7f7494a29897a3b04161857ebT7uyMLXVB
b5b331f7f7494a29897a3b04161857ebTfYkC6Wcb
b5b331f7f7494a29897a3b04161857ebtiNmtVt1v
b5b331f7f7494a29897a3b04161857ebTMUe0kIAa
b5b331f7f7494a29897a3b04161857ebtOHiNGy2j
b5b331f7f7494a29897a3b04161857ebTxchjCDDW
b5b331f7f7494a29897a3b04161857ebuByDORW0t
b5b331f7f7494a29897a3b04161857ebUtYGLUh8X
b5b331f7f7494a29897a3b04161857ebvCq5BBaZx
b5b331f7f7494a29897a3b04161857ebVHHtlmPD9
b5b331f7f7494a29897a3b04161857ebVoDniCpAm
b5b331f7f7494a29897a3b04161857ebVP2tauPxY
b5b331f7f7494a29897a3b04161857ebVqiSlaRej
b5b331f7f7494a29897a3b04161857ebW2GNrilrK
b5b331f7f7494a29897a3b04161857ebW9bEs23k8
b5b331f7f7494a29897a3b04161857ebwGHz1Y7M3
b5b331f7f7494a29897a3b04161857ebWHNHs3TO4
b5b331f7f7494a29897a3b04161857ebwJIYWzZ1W
b5b331f7f7494a29897a3b04161857ebWMzR5tK6Q
b5b331f7f7494a29897a3b04161857ebWsYNzqHXV
b5b331f7f7494a29897a3b04161857ebwttuFBdEm
b5b331f7f7494a29897a3b04161857ebwU11aWB2z
b5b331f7f7494a29897a3b04161857ebXidIPePY7
b5b331f7f7494a29897a3b04161857ebXnCBpbMja
b5b331f7f7494a29897a3b04161857ebXodBV5ex4
b5b331f7f7494a29897a3b04161857ebxrZkfUaYb
b5b331f7f7494a29897a3b04161857ebxvrAIAug0
b5b331f7f7494a29897a3b04161857eby8N2lWm88
b5b331f7f7494a29897a3b04161857ebY8nJk3W6G
b5b331f7f7494a29897a3b04161857ebylBP9VE3m
b5b331f7f7494a29897a3b04161857ebYNgknfeZB
b5b331f7f7494a29897a3b04161857ebyqMfoNe9s
b5b331f7f7494a29897a3b04161857ebYyOUK4hZR
b5b331f7f7494a29897a3b04161857ebZAxl6miVS
b5b331f7f7494a29897a3b04161857ebzmKWuV7hV
b5b331f7f7494a29897a3b04161857ebZMqOcNlGg
b5b331f7f7494a29897a3b04161857ebzn3dy0cyK
b5b331f7f7494a29897a3b04161857ebZTc45qjDx
b5b331f7f7494a29897a3b04161857ebzVRQ2JuYk

These extra directories seem to only contain a media-1 directory which contain a bunch of segment-{number}.ts files, but they don't contain m3u8 files. Only the first one which is just the media MD5 contains m3u8. I'm not sure these extra directories are actually linked to by anything since I have yet to find in the code what would cause this, so I think the extra directories could probably be deleted, but I don't know. If I run ls -l | grep -P '[0-9a-f]{32}.{9}' | wc -l to find how many of these "extra" directories I have, I get 658. That's a lot of additional space consumption if these are truly extra.

To Reproduce
Unfortunately, I have no idea how to reproduce this. Some videos have the extra directories, some don't.

Expected behavior
Not leave a bunch of junk data on the disk to take up terabytes of additional space.

Screenshots
N/A

Environment (please complete the following information):

• OS: Ubuntu Linux
• Installation method: single server install
• Browser, if applicable: N/A

Additional context
Add any other context about the problem here.

[KyleMaas]

Okay, so that's coming from here:

mediacms/files/tasks.py

Line 432 in c5047d8

output_dir = os.path.join(settings.HLS_DIR, p + produce_friendly_token())

produce_friendly_token() generates a 9-character string by default. So these are apparently temporary directories which would otherwise be deleted by:

mediacms/files/tasks.py

Line 440 in c5047d8

shutil.rmtree(output_dir)

So at this point, I'm pretty sure that these extra directories can safely be deleted.

[cphpi]

great

[foeyonghai]

Hi @KyleMaas, are you able to trace back which video in your instance did cause this? Maybe it has something to do with:

• extention
• size
• resolution
• etc...

Another thing that I am very curious to is if this maybe could be caused by a failed upload. Sometimes in MediaCMS the upload displays an error like "upload failed" but afterwards seems to succeed in processing the video. The scenario that I am thinking of is that after getting an error the cleanup actions are halted/skipped.

You mentioned that it not always happens, that is something that triggered me.

By the way, I am not one of the developers/contributers of MediaCMS but I am such a big fan that I descided to dive deeper in it, therefore I am trying to help troubleshooting in some issues.

Best regards!

[KyleMaas]

@foeyonghai Unfortunately, I haven't been able to figure out any commonalities between the videos that are exhibiting the problem. Videos which don't leave behind junk are of similar size, extension, resolution, and duration as ones which don't leave behind junk. "Upload failed" is a fairly common occurrence around here, though, although I can't say all of them that show "upload failed" have this happen or all of them that leave behind junk show "upload failed". So I don't really know. But thanks for helping troubleshoot this!