Jenkins disks filling up

[wohali]

module 'couchdb_compaction_daemon_tests'
  Compaction daemon tests
    couchdb_compaction_daemon_tests:74: should_compact_by_default_rule...*timed out*
    undefined
  couchdb_compaction_daemon_tests:105: should_compact_by_dbname_rule...*timed out*
  undefined

• https://builds.apache.org/blue/organizations/jenkins/CouchDB/detail/master/41/pipeline/41
  • https://couchdb-vm2.apache.org/ci_errorlogs/jenkins-couchdb-41-2017-06-24T14:43:39.566358/couchlog.tar.gz

We've seen this before as COUCHDB-3344 and attempts to fix it as recent as 2017.05.20 have failed. Full history:

[wohali]

[warning] 2017-06-24T14:42:21.142947Z nonode@nohost <0.6005.1> -------- Compaction daemon - skipping database `eunit-test-db-1498315211680861` compaction: the estimated necessary disk space is about 107084 bytes but the currently available disk space is 0 bytes.

x100 or so. Invalid result. Closing for now. Reopen if this recurs.

[wohali]

This is back:

https://builds.apache.org/blue/organizations/jenkins/CouchDB/detail/master/88/pipeline/49
https://builds.apache.org/blue/organizations/jenkins/CouchDB/detail/master/88/pipeline/48
https://builds.apache.org/blue/organizations/jenkins/CouchDB/detail/master/87/pipeline/41
https://builds.apache.org/blue/organizations/jenkins/CouchDB/detail/master/87/pipeline/42
https://builds.apache.org/blue/organizations/jenkins/CouchDB/detail/master/89/pipeline/41
module 'couchdb_compaction_daemon_tests'
Compaction daemon tests
couchdb_compaction_daemon_tests:75: should_compact_by_default_rule...timed out
undefined
couchdb_compaction_daemon_tests:106: should_compact_by_dbname_rule...timed out
undefined

Need to analyze the couch logs to see if we're running out of disk space.

[wohali]

Again @ https://builds.apache.org/blue/organizations/jenkins/CouchDB/detail/master/92/pipeline/41, but:

[warning] 2017-07-13T17:20:13.606448Z nonode@nohost <0.15086.1> -------- Compaction daemon - skipping database `_dbs` compaction: the estimated necessary disk space is about 53890 bytes but the currently available disk space is 0 bytes.

[wohali]

I talked with INFRA just now and handed them a pointer to https://github.com/apache/couchdb-ci/blob/master/bin/cleanup-docker.sh . They're going to put something similar in a cron job on the Jenkins build agents.