consistently on convert, invalid json returned from container

[keywordnew]

On attempting to convert one or more files, error of invalid json returned from container. This happens consistently, and no files are converted successfully.

versions

• Dangerzone: 0.4.0
• Docker: 4.17.0
• OS: MacOS 13.2.1, Intel

reproduce

1. select file(s) of type .pdf, etc. I've tried a few.
2. tap Convert to Safe Document.

other context

This issue is a change from long-time use, where conversion was working. This error began on 2023.02.06. The first indication of something being broken was Dangerzone failing to detect Docker.

I removed both, and reinstalled. This resolved Dangerzone not detecting Docker. Then invalid json returned from container began appearing.

Removing, and reinstalling has no further effect.

[apyrgio]

A user had reported a similar issue a few months ago: #253

Could you run the following commands, to make sure that Docker works properly?

sudo docker info
sudo docker run hello-world

[keywordnew]

Thanks for the suggestion.

docker info gives me:

Client:
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.10.3)
  compose: Docker Compose (Docker Inc., v2.15.1)
  dev: Docker Dev Environments (Docker Inc., v0.1.0)
  extension: Manages Docker extensions (Docker Inc., v0.2.18)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.25.0)
  scout: Command line tool for Docker Scout (Docker Inc., v0.6.0)

docker run hello-world outputs:

...
This message shows that your installation appears to be working correctly.
...

One difference against past issue. The line Context: desktop-linux under Client is different from previous issue, but that seems to be explained by "The default docker CLI context is now desktop-linux." on MacOS.

I recall coming across #253 in earlier troubleshooting. But couldn't get dangerzone in command line, then abandoned that line of debugging.

[apyrgio]

Thanks for the command output. So, Docker seems to be working correctly.

Next thing we should check is what is the message that the container throws. I'm not that well-versed on MacOS, so I don't know where the standard streams are redirected, when you double click on the command. However, you should be able to run Dangerzone from the terminal, and you should find more info there.

Run the following for the Dangerzone application:

/Applications/Dangerzone.app/Contents/MacOS/dangerzone

Or the following for the Dangerzone CLI:

/Applications/Dangerzone.app/Contents/MacOS/dangerzone-cli <filename>

[keywordnew]

Thanks for including MacOS-specific instructions to dig out further information.

steps

run:
/Applications/Dangerzone.app/Contents/MacOS/dangerzone-cli test.pdf

result

I get the following output from the CLI:

CLI output from running Dangerzone CLI against test.pdf

Assigning ID 'Z12ECd' to doc '<path to filename>'

Converting document to safe PDF
Marking doc Z12ECd as 'converting'
> /usr/local/bin/docker run --network none --platform linux/amd64 -u dangerzone --security-opt=no-new-privileges:true --cap-drop all --rm -v <path to filename>:/tmp/input_file -v '/Users/<user>/Library/Application Support/dangerzone/tmp/tmp5f34ae8_/pixels:/dangerzone' dangerzone.rocks/dangerzone /usr/bin/python3 /usr/local/bin/dangerzone.py document-to-pixels
Invalid JSON returned from container:

	 Traceback (most recent call last):

Marking doc Z12ECd as 'failed'
Invalid JSON returned from container:

	   File "/usr/local/bin/dangerzone.py", line 389, in <module>

Marking doc Z12ECd as 'failed'
Invalid JSON returned from container:

	     sys.exit(main())

Marking doc Z12ECd as 'failed'
Invalid JSON returned from container:

	   File "/usr/local/bin/dangerzone.py", line 378, in main

Marking doc Z12ECd as 'failed'
Invalid JSON returned from container:

	     converter.document_to_pixels()

Marking doc Z12ECd as 'failed'
Invalid JSON returned from container:

	   File "/usr/local/bin/dangerzone.py", line 137, in document_to_pixels

Marking doc Z12ECd as 'failed'
Invalid JSON returned from container:

	     mime_type = mime.from_file("/tmp/input_file")

Marking doc Z12ECd as 'failed'
Invalid JSON returned from container:

	   File "/usr/lib/python3.10/site-packages/magic/__init__.py", line 111, in from_file

Marking doc Z12ECd as 'failed'
Invalid JSON returned from container:

	     with _real_open(filename):

Marking doc Z12ECd as 'failed'
Invalid JSON returned from container:

	 PermissionError: [Errno 1] Operation not permitted: '/tmp/input_file'

Marking doc Z12ECd as 'failed'
documents-to-pixels failed

Failed to convert document(s)
<path to filename>

findings

1. I had a feeling that granting Dangerzone permissions for Full Disk Access might resolve Operation not permitted. I tried it. I was wrong. No effect.

2. Output from CLI looks similar to that of [Errno 13] Permission denied error on Fedora 36 #260.

   • Except: in issue 260, system is Linux, and PermissionError line specifies Errno 13 instead.
   • Anticipating next step to be similar to [Errno 13] Permission denied error on Fedora 36 #260 (comment), these are the permissions for the test file I used:
     -rw-r--r--@
     • Is the issue I'm experiencing similar to what you described in comment: Permission denied: container can't write to /dangerzone #157 (comment)?
     • If so, would it be fixed by release v0.4.1, which fixes Reliably mount input files #335?

[apyrgio]

Nice, thanks a lot for the container output! It's interesting, we have not seen it before. This error seems to be related to some permission-related changes on Big Sur: https://apple.stackexchange.com/questions/425519/why-do-i-get-ls-desktop-operation-not-permitted-when-i-own-desktop

Thanks also for the output of ls -l. I'm afraid we will need a bit more info though:

1. What is the full output of ls -le <file>? Feel free to anonymize any results, if you want.
2. What is the path of that file?
3. What is your MacOS version and CPU type (Intel, M1/2?)

We can try to reproduce this issue based on the above info. Then, we can verify if this will be fixed in 0.4.1. In the meantime, you can try it out yourself, following our build instructions for MacOS minus the Homebrew and app bundle stuff.

[apyrgio]

Nice, thanks a lot for the container output! It's interesting, we have not seen it before. This error seems to be related to some permission-related changes on Big Sur: https://apple.stackexchange.com/questions/425519/why-do-i-get-ls-desktop-operation-not-permitted-when-i-own-desktop

Thanks also for the output of ls -l. I'm afraid we will need a bit more info though:

1. What is the full output of ls -le <file>? Feel free to anonymize any results, if you want.
2. What is the path of that file?
3. What is your MacOS version and CPU type (Intel, M1/2?)

We can try to reproduce this issue based on the above info. Then, we can verify if this will be fixed in 0.4.1. In the meantime, you can try it out yourself, following our build instructions for MacOS minus the Homebrew and app bundle stuff.

[apyrgio]

(duplicate message due to some GitHub UI issues)

[keywordnew]

Thanks for including the resource on permission changes starting from Big Sur! Below, items 1 through 3 are related to prompts from #371 (comment). 4 and 5 are additional.

1. What is the full output of ls -le ? Feel free to anonymize any results, if you want.

-rw-r--r--@ 1 <owner is user> <owner group is staff> 6346712 <date> <hh:mm> test.pdf

2. What is the path of that file?

~/Downloads/test.pdf

3. What is your MacOS version and CPU type (Intel, M1/2?)

• Intel
• Added the CPU type to the first post for others who pass this way. If you see it later and wonder how you missed the info, you didn't miss it, this edit is fresh :-)

4. "error seems to be related to some permission-related changes on Big Sur: https://apple.stackexchange.com/questions/425519/why-do-i-get-ls-desktop-operation-not-permitted-when-i-own-desktop"

• Had a look at the stackexchange. My interpretation is that enabling Full Disk Access for Dangerzone and/or Docker should have an effect.
• Earlier, I gave Dangerzone this permission, while writing consistently on convert, invalid json returned from container #371 (comment), which had no effect.
• This time, I gave Docker the Full Disk Access permission. Conversion completes successfully from GUI and CLI.
  • As before, below is the container output for a successful conversion with the fix in place, FWIW.

CLI output from running Dangerzone CLI against test.pdf with Full Disk Access granted to Docker

Assigning ID 'kuoTc_' to doc '<path to filename>'

Converting document to safe PDF
Marking doc kuoTc_ as 'converting'
> /usr/local/bin/docker run --network none --platform linux/amd64 -u dangerzone --security-opt=no-new-privileges:true --cap-drop all --rm -v <path to filename>:/tmp/input_file -v '/Users/<user>/Library/Application Support/dangerzone/tmp/tmp7mujq9c3/pixels:/dangerzone' dangerzone.rocks/dangerzone /usr/bin/python3 /usr/local/bin/dangerzone.py document-to-pixels
[doc kuoTc_] 3% Separating document into pages
[doc kuoTc_] 5% Converting page 1/2 to pixels
[doc kuoTc_] 27% Converting page 2/2 to pixels
[doc kuoTc_] 50% Converted document to pixels
> /usr/local/bin/docker run --network none --platform linux/amd64 -u dangerzone --security-opt=no-new-privileges:true --cap-drop all --rm -v '/Users/<user>/Library/Application Support/dangerzone/tmp/tmp7mujq9c3/pixels:/dangerzone' -v '/Users/<user>/Library/Application Support/dangerzone/tmp/tmp7mujq9c3/safe:/safezone' -e OCR=0 -e OCR_LANGUAGE=None dangerzone.rocks/dangerzone /usr/bin/python3 /usr/local/bin/dangerzone.py pixels-to-pdf
[doc kuoTc_] 50% Converting page 1/2 from pixels to PDF
[doc kuoTc_] 72% Converting page 2/2 from pixels to PDF
[doc kuoTc_] 95% Merging 2 pages into a single PDF
[doc kuoTc_] 97% Compressing PDF
[doc kuoTc_] 100% Safe PDF created
Marking doc kuoTc_ as 'safe'

Safe PDF(s) created successfully

5. I don't know if related, but at the same time as this issue, I've been experiencing intermittent occurrences of docker is not installed.

• Looking through my notes, I see that this issue (consistently on convert, invalid json returned from container #371) was immediately preceded by an incorrect instance of Docker not being detected. Possibly the first of these incorrect detection failures, but I can't be sure.
• As I did to fix docker is not installed today, at that time I fixed the error by removing and reinstalling Docker.
• Do you think unreliable detection of Docker is related to the permissions issue?

[apyrgio]

This time, I gave Docker the Full Disk Access permission. Conversion completes successfully from GUI and CLI.

Yay! But also, damn. It worries me that more and more users will start stumbling on this issue, and these users will not be tech-savvy :-/. Ultimately, that is a Docker Desktop problem, so hopefully they will offer a fix as part of the installation. Else, any docker run -v command will fail.

Do you think unreliable detection of Docker is related to the permissions issue?

To the best of my knowledge, no. Funny thing though is that at the same time you posted this issue, another user posted a problem related to Docker not being found: #374. They resolved it in a brute kind of way, but maybe something in this thread helps you understand what's going on.

[keywordnew]

Further finding on minimum permissions required: FWIW it's enough to give Docker access to only the directory containing the intended file(s) to be converted. Full Disk is unnecessary.

Funny thing though is that at the same time you posted this issue, another user posted a problem related to Docker not being found: #374

Thanks for linking to a relevant conversation. To avoid cluttering this thread with a likely unrelated problem, I'll tackle it separately.

worries me that more and more users will start stumbling on this issue, and these users will not be tech-savvy

My research shows that apps which depend on similar permissions, to operate as expected, display a modal to inform users about the suggested settings. They may also choose to include a button that surfaces the relevant view in System Settings, for Privacy & Security. Is something similar useful/actionable enough for users here?

[apyrgio]

FWIW it's enough to give Docker access to only the directory containing the intended file(s) to be converted. Full Disk is unnecessary.

Nice, highlighting the important part so that other users notice it as well.

As for showing a modal to the user for System Integrity Protection (SIP): I would expect that this is something that Docker Desktop would have to do, whenever it detected that a user wants to mount a path that it doesn't have permissions for. I'm afraid I don't see how Dangerzone can trigger such a modal for Docker, given that csrutil seems to either globally enable or disable SIP. If you do know of a way to trigger such a modal though, let us know.

[apyrgio]

Regardless of what Dangerzone can do in subsequent versions, let's settle on some next steps to resolve this issue. My suggestion is to add a note about this error on our README ("I get invalid json returned from container on MacOS Big Sur onwards") , and a short description of how users can circumvent it. Since you have done this procedure recently, would you mind sending a PR with the steps you followed? Else, I'll see if I can draft something.