You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Qubes converter currently handles /tmp in an inefficient way, leading to ENOSPC situations, when a document has more than 160 US letter pages.
Background
We have a known problem regarding the way we store pixel data in tmpfs (see #526), which becomes even worse in Qubes. The reason is that Qubes mounts tmpfs under /tmp this way:
tmpfs on /tmp type tmpfs (rw,size=1048576k,nr_inodes=35349,inode64)
The size of the /tmp dir in app and disposable qubes is therefore max 1GiB. At the same time, pdftoppm converts the pages at 150 DPI, which means each RGB page requires ~6.2MiB. Therefore, we can store at most ~165 pages in /tmp.
This limitation manifests both at the server-side, and at the client side. In the server-side, pdftoppm writes PPM files under /tmp/page*. We have a callback that writes the final RGB buffer to stdout and then removes the original PPM file, but these two processes are not synchronized, especially if the client-side does not catch up:
In the client-side, we immediately write the received pixel data under /tmp/dangerzone, since this is where the pixels-to-pdf convert expects to find them:
We don't have a solid solution yet. If we fix the client-side issue, by converting the RGB files to PNG immediately, then the disposable qube can still run out of space, if the client-side does not catch up (e.g., due to the conversion). We need a solution that fixes both sides.
The text was updated successfully, but these errors were encountered:
deeplow
changed the title
Qubes: Conversion of medium-sized documents fills up /tmp
(server) Qubes: Conversion of medium-sized documents fills up /tmp
Oct 24, 2023
PDFtoPPM was producing RGB files faster than they were getting consumed.
Since the RGB files were only getting removed after they were sent, this
was leading to /tmp in the server getting clogged.
This solution consists in processing and sending images in chunks of 50
pages. This solution is slightly inneficient since it can't process and
send data simultaneously. That will be solved in a future commit.
Fixes#574
PDFtoPPM was producing RGB files faster than they were getting consumed.
Since the RGB files were only getting removed after they were sent, this
was leading to /tmp in the server getting clogged.
This solution consists in processing and sending images in chunks of 50
pages. This solution is slightly inefficient since it can't process and
send data simultaneously. That will be solved in a future commit.
Fixes#574
I have confirmed now that with #627 the server doesn't run out of space. It does however still fail on the 164th page because we haven't yet addressed the client-side limitation #526. This will be easier anyways when we look at #625.
Problem
The Qubes converter currently handles
/tmp
in an inefficient way, leading to ENOSPC situations, when a document has more than 160 US letter pages.Background
We have a known problem regarding the way we store pixel data in
tmpfs
(see #526), which becomes even worse in Qubes. The reason is that Qubes mountstmpfs
under/tmp
this way:The size of the
/tmp
dir in app and disposable qubes is therefore max 1GiB. At the same time,pdftoppm
converts the pages at 150 DPI, which means each RGB page requires ~6.2MiB. Therefore, we can store at most ~165 pages in/tmp
.This limitation manifests both at the server-side, and at the client side. In the server-side,
pdftoppm
writes PPM files under/tmp/page*
. We have a callback that writes the final RGB buffer to stdout and then removes the original PPM file, but these two processes are not synchronized, especially if the client-side does not catch up:dangerzone/dangerzone/conversion/doc_to_pixels.py
Lines 336 to 342 in c4fdebc
In the client-side, we immediately write the received pixel data under
/tmp/dangerzone
, since this is where the pixels-to-pdf convert expects to find them:dangerzone/dangerzone/isolation_provider/qubes.py
Lines 132 to 144 in c4fdebc
Solution
We don't have a solid solution yet. If we fix the client-side issue, by converting the RGB files to PNG immediately, then the disposable qube can still run out of space, if the client-side does not catch up (e.g., due to the conversion). We need a solution that fixes both sides.
The text was updated successfully, but these errors were encountered: