New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disk full in /tmp, dup-xxxx files not getting cleaned up #3652

Open
instantlinux opened this Issue Feb 8, 2019 · 4 comments

Comments

Projects
None yet
3 participants
@instantlinux
Copy link

instantlinux commented Feb 8, 2019

  • [x ] I have searched open and closed issues for duplicates.

Environment info

  • Duplicati version: 2.0.4.2_experimental_2018-11-12
  • Operating system: Debian Stretch (docker)
  • Backend: Backblaze B2

Description

Error while running <backup name>
Disk full. Path /tmp/dup-30a88439-df78-4a2e-8840-5763ee2b27df

Steps to reproduce

  1. Set up a backup
  2. Run for a couple of weeks
  • Actual result:
    Backup didn't complete
  • Expected result:
    The /tmp volume should be cleaned up after each run

Screenshots

10GB of tmp files created in past 2 weeks:

# ls -l
total 9229696
drwxr-xr-x 2 root backup      4096 Jan 26 12:55 HttpServer
-rw-r--r-- 1 root backup 524304366 Feb  4 07:24 dup-0d5f37bd-af00-4135-98ca-e0a3b2459967
-rw-r--r-- 1 root backup       156 Jan 28 02:06 dup-0ee35bc2-01eb-4a5f-9d49-6324b44f4d56
-rw-r--r-- 1 root backup 524319268 Feb  4 06:34 dup-137e3a87-fdfa-4473-b443-adf27fd46e95
-rw-r--r-- 1 root backup    555615 Feb  4 06:24 dup-1726a950-3ad2-453e-9940-526655ca74c4
-rw-r--r-- 1 root backup 524351678 Jan 30 09:49 dup-1e897042-35ca-4850-8050-656d950d7b10
-rw-r--r-- 1 root backup       338 Feb  7 03:53 dup-20bb8b9f-0df9-4162-bf58-5d9bfd51f6f0
-rw-r--r-- 1 root backup 524259161 Feb  7 03:53 dup-20c1038c-952f-4cdd-a510-e95c0b695ff1
-rw-r--r-- 1 root backup    299676 Feb  6 09:15 dup-2186b888-72ce-41d6-9ce5-bb107a78eaa9
-rw-r--r-- 1 root backup       156 Jan 29 03:01 dup-21ca1928-d0e2-4888-9d75-ad1e0fcad742
-rw-r--r-- 1 root backup       156 Feb  4 09:35 dup-2293a43b-0768-40d2-9c9f-12bf4f54b4ad
-rw-r--r-- 1 root backup       338 Feb  2 03:10 dup-24c0a8f7-298e-4c9c-9a58-3abd9355a5cb
-rw-r--r-- 1 root backup    473517 Feb  4 06:36 dup-277c81fd-cc3c-40fa-9836-33f2c0cfe006
-rw-r--r-- 1 root backup       338 Feb  5 03:32 dup-28197084-669e-4b34-b7d7-bf2af833e350
-rw-r--r-- 1 root backup       338 Feb  6 02:38 dup-2f690ef8-5547-416b-bdf5-ae9e6f4fb2b3
-rw-r--r-- 1 root backup  32915456 Feb  7 03:53 dup-30a88439-df78-4a2e-8840-5763ee2b27df
-rw-r--r-- 1 root backup       336 Feb  7 02:37 dup-30ca7aeb-1013-41d9-9f85-7c4c7ac0b565
-rw-r--r-- 1 root backup       156 Jan 29 06:12 dup-331cd9b7-64b5-4ac2-927e-fb85c603764e
-rw-r--r-- 1 root backup  44161045 Feb  2 02:10 dup-35f90cc2-58c9-4820-a03d-ea7c0c21802d
-rw-r--r-- 1 root backup 524385525 Feb  4 05:16 dup-36a6a7a3-dbec-4cc3-a240-612c6e74808d
-rw-r--r-- 1 root backup 354289965 Feb  7 03:50 dup-38f9f7b9-9e93-4d53-9e51-97ea6b08d9b2
-rw-r--r-- 1 root backup    288059 Jan 30 10:23 dup-39296066-e183-4290-afd8-b514e2a447a1
-rw-r--r-- 1 root backup    223523 Feb  4 07:16 dup-41912b2a-9393-4889-9f64-57ca778a167e
-rw-r--r-- 1 root backup 524300770 Feb  4 06:59 dup-48adda0b-ab45-4c0d-b038-3f3aebbac6c4
-rw-r--r-- 1 root backup       154 Jan 28 07:45 dup-49e7185d-ffee-43da-a1c4-b3f2322f6d1f
-rw-r--r-- 1 root backup 524359007 Feb  4 06:04 dup-511e3831-7d5e-4c25-b335-9360f04455ba
-rw-r--r-- 1 root backup   5047757 Feb  4 07:02 dup-5372311a-8226-4d44-8517-6651c8802690
-rw-r--r-- 1 root backup       231 Feb  5 10:01 dup-5ec5cfbe-28eb-41c7-9fb8-944f5be644cc
-rw-r--r-- 1 root backup 524288136 Feb  4 06:22 dup-5fa57ae1-3c0e-42de-887b-f34a6ae66402
-rw-r--r-- 1 root backup 251440192 Feb  6 09:15 dup-6431ce67-d394-4d73-aff6-b00fa4649ecb
-rw-r--r-- 1 root backup    405267 Jan 30 09:52 dup-650df6cd-07e5-4e4c-a4b6-5362e0a632b4
-rw-r--r-- 1 root backup    405333 Feb  4 05:19 dup-7134b5bd-e62b-4b88-898f-2caea81c2f21
-rw-r--r-- 1 root backup       248 Feb  4 09:35 dup-77610af4-73f9-40d4-b60b-89afbdd25bb3
-rw-r--r-- 1 root backup       336 Feb  4 03:05 dup-78e0fd58-fc95-4fac-b4aa-62ea9bacf4c9
-rw-r--r-- 1 root backup 524356993 Feb  4 07:57 dup-7c2ff270-98f4-4758-926d-28d571b81bd3
-rw-r--r-- 1 root backup    287818 Feb  4 08:17 dup-871d7d2e-9745-4fff-bd6f-6918e0720c17
-rw-r--r-- 1 root backup       248 Jan 29 06:13 dup-8868a6ec-c227-4685-a14d-8caaac19d08e
-rw-r--r-- 1 root backup 524318973 Feb  4 07:34 dup-89516df7-dc66-422c-9bea-c07301bce677
-rw-r--r-- 1 root backup       248 Jan 28 02:06 dup-8b158d44-62f7-4765-9d79-9ee81d391229
-rw-r--r-- 1 root backup       336 Jan 29 03:18 dup-8d096ca3-fe0d-4eef-82bc-2227966805e6
-rw-r--r-- 1 root backup    589830 Feb  4 08:28 dup-956958b9-3feb-4862-b3d4-ba3903e305fe
-rw-r--r-- 1 root backup 524349975 Feb  4 07:46 dup-9a23a028-249f-4d8e-8315-7c228a2e3b52
-rw-r--r-- 1 root backup 524345890 Feb  4 05:51 dup-9dc987e0-20e3-47bb-8faf-79c70e00b3de
-rw-r--r-- 1 root backup    237374 Feb  4 07:37 dup-9e242a7e-2cf4-4fc7-8889-6d0355561823
-rw-r--r-- 1 root backup 524360222 Feb  4 07:13 dup-a00a1db3-4071-4fa0-af27-0c73993f2ecd
-rw-r--r-- 1 root backup    363183 Feb  4 05:56 dup-a5c554d7-9b4e-43f2-a2bf-19e26fd76c35
-rw-r--r-- 1 root backup       246 Jan 28 07:45 dup-a9342b92-c9d8-4f54-9437-5ce71ce65e80
-rw-r--r-- 1 root backup    176248 Feb  4 07:26 dup-ac44d748-7470-4dad-99d8-4917b6e397f0
-rw-r--r-- 1 root backup    305084 Feb  4 06:06 dup-afe39750-d80e-4fc2-a443-de439160a637
-rw-r--r-- 1 root backup       334 Feb  1 03:07 dup-b34a9dc0-92ff-4a5b-8c98-d46ba3ed6f1e
-rw-r--r-- 1 root backup       338 Jan 31 02:40 dup-ba784021-cabd-40b1-b4b0-8e32e33392fc
-rw-r--r-- 1 root backup       231 Feb  5 10:01 dup-ba93a6df-4416-4efe-bf62-e7510d86e8fd
-rw-r--r-- 1 root backup    243140 Feb  4 08:00 dup-bdc2143c-aeb2-482e-89b0-96fccf9e8304
-rw-r--r-- 1 root backup       248 Jan 29 03:01 dup-bf182425-3849-49c6-8d3b-ac247b0c66aa
-rw-r--r-- 1 root backup 524305195 Feb  4 05:30 dup-d306e6dd-ced1-4b56-8276-668d19ce8e1e
-rw-r--r-- 1 root backup 524354359 Feb  4 08:14 dup-d49eceda-bb90-45d4-8758-a6a9e51721ab
-rw-r--r-- 1 root backup    195619 Feb  4 05:34 dup-dd831796-7703-4ee4-a31d-2cfe7b941086
-rw-r--r-- 1 root backup       336 Jan 30 03:09 dup-e32ee3af-6fbb-436e-88d5-9b497bc13fa3
-rw-r--r-- 1 root backup    217456 Feb  4 07:48 dup-e7a9f9d3-2521-494b-a181-acf4c6df21ff
-rw-r--r-- 1 root backup       337 Feb  3 03:10 dup-ecc3ccf2-d3ce-4876-aa67-a692bb85119d
-rw-r--r-- 1 root backup 367331881 Jan 30 10:22 dup-ed7a5f15-6ed2-449c-81ab-f763810fab0e
-rw-r--r-- 1 root backup 524355343 Feb  4 08:28 dup-f09b3d7f-8852-42bb-a263-8ffbd5eff129
-rw-r--r-- 1 root backup   1163860 Feb  2 02:10 dup-fb9f3e3b-a30a-43af-b90c-a51e6cc7df19
-rw-r--r-- 1 root backup         0 Feb  7 03:53 dupl-usagereport-239-20190207113222.json

Debug log

N/A

@warwickmm

This comment has been minimized.

Copy link
Contributor

warwickmm commented Feb 8, 2019

Related discussion here.

@instantlinux

This comment has been minimized.

Copy link
Author

instantlinux commented Feb 8, 2019

Yes, I saw that discussion. Somehow it never got filed as an actual bug.

@warwickmm

This comment has been minimized.

Copy link
Contributor

warwickmm commented Feb 8, 2019

One possible explanation may be forgetting to dispose of instances of BlockVolumeWriter. It appears that this object contains a TempFile, which represents a file that is deleted upon disposal. The discussion linked above mentions files being left over from a compact operation. Looking at the CompactHandler code, it appears that we use a BlockVolumeWriter without wrapping it in a using block:

https://github.com/duplicati/duplicati/blob/master/Duplicati/Library/Main/Operation/CompactHandler.cs#L104

@jheyneman

This comment has been minimized.

Copy link

jheyneman commented Feb 8, 2019

Perhaps this is related to issue #3634 , my ramdisk was filling up and the number of temp archives to create is exceeded.

warwickmm added a commit to warwickmm/duplicati that referenced this issue Feb 11, 2019

Ensure removal of VolumeWriterBase temporary files.
Each VolumeWriterBase contains a TempFile, which deletes a file upon
disposal.  If we do not dispose of this instance, temporary files may be
left behind.

Some of these instances could not be put inside a using block, as the
variables are being redefined inside a loop (a using block marks the
object as readonly).

This concerns issue duplicati#3652.

warwickmm added a commit to warwickmm/duplicati that referenced this issue Feb 17, 2019

Ensure that FilesetVolumeWriter removes temp file upon disposal.
This ensures the removal of the temporary file containing the
filelist.json that remained regardless if files needed to be uploaded or
not.

This concerns issue duplicati#3652.

warwickmm added a commit to warwickmm/duplicati that referenced this issue Feb 17, 2019

warwickmm added a commit to warwickmm/duplicati that referenced this issue Feb 17, 2019

Allow removal of VolumeWriterBase temp files.
This reverts a change made in revision 31ac400 ("Prevent temp files
in VolumeWriterBase from being deleted prematurely" in pull request duplicati#2637).
It's unclear what issue was being worked around by preventing the
removal of the temporary files (based on comments in pull request duplicati#2637,
it appears to be related to tests that failed only on Travis).  However,
this workaround prevented the removal of temporary files created during
compact operations, which caused several users' temporary directories to
be filled up with files that should have been removed.

This concerns issue duplicati#3652.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment