Option to remove blank pages when importing documents #668
-
|
I'd like to have to option to automatically remove blank pages when importing a document. This correlates to #528 where the possiblity to remove pages manually is requested. |
Beta Was this translation helpful? Give feedback.
Replies: 14 comments 46 replies
-
|
https://github.com/baltpeter/scanprep |
Beta Was this translation helpful? Give feedback.
-
|
maybe a combination of PAPERLESS_PRE_CONSUME_SCRIPT and this approach will do the trick. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Mates, I use the docker-setup would like to give u brief overview of my solution:
Features
|
Beta Was this translation helpful? Give feedback.
-
|
I've edited my answer. Thx @r748 |
Beta Was this translation helpful? Give feedback.
-
hey @username914 and @KelvinGradCelsius , |
Beta Was this translation helpful? Give feedback.
-
|
No I wasn't yet able to fix and still hope it'll become someday part of the official fork. |
Beta Was this translation helpful? Give feedback.
-
|
I am getting a different failure. For my docker installation no pdfinfo is installed. Any good tips where to get support intalling ist on docker.? |
Beta Was this translation helpful? Give feedback.
-
|
Is there any more information about the value of Threshold and how I get the correct value of it? Instead of dumb try and error method. |
Beta Was this translation helpful? Give feedback.
-
|
In my eyes this could be a great feature since for instance I always have to split one sided and double sided documents before scanning and even though the mixed docs will have the blank pages still. I wasn't able to install the script solution from @psi-4ward on my QNAP Docker installation. May it could make it's way to the roadmap. Would be great. Paperless is really awesome 👍 thanks. |
Beta Was this translation helpful? Give feedback.
-
|
Great feature and great idea :) |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately, I haven't been able to get the script to work. Can anyone help me out? Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
I think inkcov is simply not really suited for the task.
I believe ink_cov is the latter and better suited for detecting blank pages. |
Beta Was this translation helpful? Give feedback.
-
|
ink_cov seems to be a bit more consistent, so thanks to @felixgonschorek for this hint. Will try to to use that one... |
Beta Was this translation helpful? Give feedback.
-
|
Did some tests/work yesterday and i am happy with the version of @felixgonschorek, I adjusted the theshold to my needs (scan a white page and see which value is produced by gs, and go slightly above it not to accidentally remove pages). So for my Canon MX925 0.8 seems to be fine. |
Beta Was this translation helpful? Give feedback.
-
|
@felixgonschorek I just recognized that this way with with pre-consuming only works with PDFs, or the other way round, currently a cannot import XLS oder other formats, as it seems (i use tika). Did you also encounter that? Maybe i have to extend the script to jump over non-PDFs?
|
Beta Was this translation helpful? Give feedback.
-
|
I only use paperless-ngx for PDF documents, so yes, I think you have to extend the script to ignore non-pdf files :) |
Beta Was this translation helpful? Give feedback.
-
|
Here is my script which uses ink_cov and with a threshold suitable for my scanner and use cases. Also I added some output to stderr to see which threshold was detected on added or removed pages: |
Beta Was this translation helpful? Give feedback.
-
|
I have changed the script to only delete blank pages when it is a singlesided document. For documents which are doublesided I want to keep any blank pages to keep the original page ordering. |
Beta Was this translation helpful? Give feedback.
-
|
Regarding #668 (reply in thread): Updated version from @felixgonschorek which checks for file format and skips all non PDF files. |
Beta Was this translation helpful? Give feedback.
-
|
Yes you write it: less but the sign is more. |
Beta Was this translation helpful? Give feedback.
-
|
Ok i see. Keep in mind that this line is from the debug output of bash itself (enforced by set -x), so bash itself reports what is going on, and there the check being done is indeed 0>0.3 - which fails and results in the "real" logging output "Page removed from document". |
Beta Was this translation helpful? Give feedback.
-
|
Sorry its me again: I have a 14 page document there are 2 black pages in. |
Beta Was this translation helpful? Give feedback.
-
|
Well, i already wondered why the former example (with the 2 pages) suddenly should have worked, as that one already had the same "issue" as this current example, pdfinfo not being able to find the page count. Maybe you can check the file for conformity with other pdf tools (online). So it is hard to say what happens from remote unless you provide some more details, as already being asked (output of manual pdfinfo or sample pages). At least it works fine for me (and others) with white pages :). Producer: iOS Version 16.3.1 (Build 20D67) Quartz PDFContext Writing that, i was wondering myself what the expected behaviour should be: Originally, the script has been written to remove "blank" pages, which means white pages with few (or none) "letters" on it, measuring this by amount of ink on that page. What does it mean to have a "black page"? Is it a (white) page being fully inked, or is it black background. I'm not sure what will happen then... What would you expect to get - black pages should be kept or removed? |
Beta Was this translation helpful? Give feedback.
-
|
I use |
Beta Was this translation helpful? Give feedback.
-
|
I personally have not tried any of these solutions. However, I'd like to warn that the real solution should actually delete the page contents. I know there are some PDF tools (like this) that can reorder or delete pages, but they just remove them from the index/list of pages, they do not delete the real content (it's still in there, just not visible anymore, but can still be extracted, and still takes up space). I hope the |
Beta Was this translation helpful? Give feedback.
-
|
Regarding qpdf, i am not that sure but i would rely on/expect it. At least, it works fine for me, as it is more as a convenience thing, "keeping the look somehwat smarter" for paperless' ocr. |
Beta Was this translation helpful? Give feedback.
-
|
For those who are searching for the final script, with checking all pages in a PDF. Please find my working script below which is a sum of latest comments and improvements. #!/bin/bash
#set -x -e -o pipefail
set -e -o pipefail
export LC_ALL=C
#IN="$1"
IN="$DOCUMENT_WORKING_PATH"
# Check for PDF format
TYPE=$(file -b "$IN")
if [ "${TYPE%%,*}" != "PDF document" ]; then
>&2 echo "Skipping $IN - non PDF [$TYPE]."
exit 0
fi
# PDF file - proceed
#PAGES=$(pdfinfo "$IN" | grep ^Pages: | tr -dc '0-9')
PAGES=$(pdfinfo "$IN" | awk '/Pages:/ {print $2}')
>&2 echo Total pages $PAGES
# Threshold for HP scanners
# THRESHOLD=1
# Threshold for Canon MX925
THRESHOLD=1
non_blank() {
for i in $(seq 1 $PAGES) ; do
PERCENT=$(gs -o - -dFirstPage=${i} -dLastPage=${i} -sDEVICE=ink_cov "${IN}" | grep CMYK | nawk 'BEGIN { sum=0; } {sum += $1 + $2 + $3 + $4;} END { printf "%.5f\n", sum } ')
>&2 echo -n "Color-sum in page $i is $PERCENT: "
if awk "BEGIN { exit !($PERCENT > $THRESHOLD) }"; then
echo $i
>&2 echo "Page added to document"
else
>&2 echo "Page removed from document"
fi
done
}
NON_BLANK=$(non_blank)
if [ -n "$NON_BLANK" ]; then
NON_BLANK=$(echo $NON_BLANK | tr ' ' ",")
qpdf "$IN" --replace-input --pages . $NON_BLANK --
fi |
Beta Was this translation helpful? Give feedback.
-
|
I think you messed up line 33 a little bit. I needed to replace |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, that was a copy past error (maybe too long line?). Just corrected it above ... |
Beta Was this translation helpful? Give feedback.
-
|
It would be great to either edit these scripts in the front end or get an entry in options to exclude blank pages by default. |
Beta Was this translation helpful? Give feedback.
-
|
In my experience, deleting blank pages often isn't sufficient as the reverse side of a document frequently contains irrelevant information or graphics. It would be great if users could decide which pages to delete through a user interface. |
Beta Was this translation helpful? Give feedback.
-
|
Would be great if this would be possible to decide if and which pages could be deleted |
Beta Was this translation helpful? Give feedback.
-
|
This discussion has been automatically closed because it was marked as answered. |
Beta Was this translation helpful? Give feedback.
-
|
This discussion has been automatically locked since there has not been any recent activity after it was closed. Please open a new discussion for related concerns. |
Beta Was this translation helpful? Give feedback.
and others
Hi Mates,
I use the docker-setup would like to give u brief overview of my solution:
scriptsnearby thedocker-compose.yaml- ./scripts:/scripts:roto thevolumes:section of paperlessPAPERLESS_PRE_CONSUME_SCRIPT=/scripts/pre-consume.shto the environment (ie env-file or environment block in docker-compose.yaml)scripts/pre-consume.shwithscripts/remove-blank-pages.sh