If no space left then remove old cache

[dosuken123]

Hi from GitLab

Currently, we're experiencing that Carrierwave raises Errno::ENOSPC when caching files. This is because inodes are full in the cache directory regardless of that its capacity has still some room. (See more https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5217) We manually removed old cache files/directories and the situation was mitigated, but this could happen in the future again, and ideally this recovery process should be automated.

/cc @stanhu

[dosuken123]

cc @stanhu

[dosuken123]

One concern is that when EMLINK or ENOSPC was raised, CarrierWave tries to remove cache files older than 10 minutes. This is hard coded.

I'm not sure if this value is proper for gitlab.com though, maybe it's better to parameterize this value via overriding a method e.g.

def cache_retention_policy
  10.days
end

[dosuken123]

cc @ayufan @amatsuda

[mshibuya]

Awesome, thanks!