Uploading PDF stuck

[AnarchyCent]

When uploading a PDF file, the file is stuck in this state:

I have the latest version of paperless-ng running in a Docker container on a Synology NAS DS918+.

The installation was done with Docker compose using portainer and then the command "python3 manage.py createsuperuser" was executed in the container.
docker-compose.portainer.yml:

# docker-compose file for running paperless from the Docker Hub.
# This file contains everything paperless needs to run.
# Paperless supports amd64, arm and arm64 hardware.
#
# All compose files of paperless configure paperless in the following way:
#
# - Paperless is (re)started on system boot, if it was running before shutdown.
# - Docker volumes for storing data are managed by Docker.
# - Folders for importing and exporting files are created in the same directory
#   as this file and mounted to the correct folders inside the container.
# - Paperless listens on port 8010.
#
# In addition to that, this docker-compose file adds the following optional
# configurations:
#
# - Instead of SQLite (default), PostgreSQL is used as the database server.
#
# To install and update paperless with this file, do the following:
#
# - Open portainer Stacks list and click 'Add stack'
# - Paste the contents of this file and assign a name, e.g. 'Paperless'
# - Click 'Deploy the stack' and wait for it to be deployed
# - Open the list of containers, select paperless_webserver_1
# - Click 'Console' and then 'Connect' to open the command line inside the container
# - Run 'python3 manage.py createsuperuser' to create a user
# - Exit the console
#
# For more extensive installation and update instructions, refer to the
# documentation.

version: "3.4"
services:
  broker:
    image: redis:6.0
    restart: unless-stopped

  db:
    image: postgres:13
    restart: unless-stopped
    volumes:
      - pgdata:/var/lib/postgresql/data
    environment:
      POSTGRES_DB: paperless
      POSTGRES_USER: paperless
      POSTGRES_PASSWORD: paperless

  webserver:
    image: jonaswinkler/paperless-ng:latest
    restart: unless-stopped
    depends_on:
      - db
      - broker
    ports:
      - 8010:8000
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000"]
      interval: 30s
      timeout: 10s
      retries: 5
    volumes:
      - data:/usr/src/paperless/data
      - media:/usr/src/paperless/media
      - /volume1/Dateien/Documents/ScannedDocuments:/usr/src/paperless/export
      - /volume1/Dateien/Documents/ToBeScanned:/usr/src/paperless/consume
    environment:
      PAPERLESS_REDIS: redis://broker:6379
      PAPERLESS_DBHOST: db
# The UID and GID of the user used to run paperless in the container. Set this
# to your UID and GID on the host so that you have write access to the
# consumption directory.
      USERMAP_UID: 1026
      USERMAP_GID: 100
# Additional languages to install for text recognition, separated by a
# whitespace. Note that this is
# different from PAPERLESS_OCR_LANGUAGE (default=eng), which defines the
# language used for OCR.
# The container installs English, German, Italian, Spanish and French by
# default.
# See https://packages.debian.org/search?keywords=tesseract-ocr-&searchon=names&suite=buster
# for available languages.
      #PAPERLESS_OCR_LANGUAGES: tur ces
# Adjust this key if you plan to make paperless available publicly. It should
# be a very long sequence of random characters. You don't need to remember it.
      #PAPERLESS_SECRET_KEY: change-me
# Use this variable to set a timezone for the Paperless Docker containers. If not specified, defaults to UTC.
      PAPERLESS_TIME_ZONE: Europe/Zurich
# The default language to use for OCR. Set this to the language most of your
# documents are written in.
      PAPERLESS_OCR_LANGUAGE: deu

volumes:
  data:
  media:
  pgdata:

I hope you can help me.

[iwconfig]

Hello,

please select the pasted content of your docker-compose.portainer.yml and press TAB two times (2) to indent it with 4 whitespace characters. This is syntax for code block in the Markdown language which is used for formatting here on Github.

[iwconfig]

What does the log say? Please paste it here by running

docker logs <container ID of jonaswinkler/paperless-ng:latest>

and remember to select the pasted text and press TAB twice.

[AnarchyCent]

Thanks for following up on this.
I copied only the last two blocks because they are repetitive.

ERRORS:
?: PAPERLESS_CONSUMPTION_DIR is not writeable
    HINT: Set the permissions of 
d--------- /usr/src/paperless/src/../consume
 to be writeable by the user running the Paperless services
2021-07-28 11:38:35,830 INFO spawned: 'scheduler' with pid 5362
2021-07-28 11:38:36,080 INFO exited: consumer (exit status 1; not expected)
2021-07-28 11:38:37,084 INFO spawned: 'consumer' with pid 5365
2021-07-28 11:38:37,085 INFO success: scheduler entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
SystemCheckError: System check identified some issues:

ERRORS:
?: PAPERLESS_CONSUMPTION_DIR is not writeable
    HINT: Set the permissions of 
d--------- /usr/src/paperless/src/../consume
 to be writeable by the user running the Paperless services
2021-07-28 11:38:38,337 INFO success: consumer entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-07-28 11:38:38,337 INFO exited: scheduler (exit status 1; not expected)

[AnarchyCent]

I have given full permission to the "consume" and "export" folders. Now it works, thank you very much.

Unfortunately I get the same result with a document as with the "synOCR" solution, that's why I actually wanted to switch to Paperless-NG.

I have changed the following settings:
PAPERLESS_OCR_MODE=redo
PAPERLESS_OCR_CLEAN=clean

Also I found the following command in the documentation:
"ocrmypdf --optimize 3"

However, I do not know where to enter this.

Can you give me a suggestion, or should I open a new issue?

[iwconfig]

Sorry, it's out of my scope unfortunately. I too have some bad OCR and I don't know (yet) how to fine tune it. I think it may have to do with Tesseract, which OCRmyPDF use for OCR.

I find ocrmypdf/OCRmyPDF#316 pretty interesting to read, maybe you'll find something useful there.

Regarding ocrmypdf --optimize 3,

You can pass anything you want to ocrmypdf by setting PAPERLESS_OCR_USER_ARGS=<json> in your docker-compose.env file (because you're using docker, otherwise it would be paperless.conf).

Have a look at the last subheading of "OCR settings"

[AnarchyCent]

Thanks for the hint to the "PAPERLESS_OCR_USER_ARGS="!

I have already read this post and unfortunately did not find anything useful in it. I have also tried many settings in the "OCR Settings" without success.

However, I read an exciting article by a guy who had to make scanned documents searchable for the Malaysian government.
https://kaerumy.medium.com/cleaning-up-scanned-documents-with-open-source-tools-9d87e15305b
He first converted the PDF's into images, cleaned the images and then converted the images back into PDF format.
It would be great to have this feature out-of-the-box with Paperless-NG as well.

[iwconfig]

Paperless-ng does this by default. This is what unpaper is for, and OCRmyPDF use it under the hood to deskew, rotate and clean the pages.

Did you try setting --optimize 3 with PAPERLESS_OCR_USER_ARGS="{'optimize': 3}"? What about --oversample <DPI>?

By the way, that document of yours, is it a scanned document or a photo? Have you read the man page of Unpaper? You can really fine tune color, brightness and stuff with it. Your screenshot seem kind of dark and murky, and maybe not completely visible for Tesseract (I would think).

Run man unpaper for the manual, or check it out here. You can set arguments to unpaper by with PAPERLESS_OCR_USER_ARGS="{'unpaper_args': ['--foo bar', '--baz qux']}" (I would assume a list is needed for multiple arguments)