[Bug]: Images Larger than a few hundred KBs fail to upload.

[EuphoricPenguin]

Requirements

• Is this a bug report? For questions or discussions use https://lemmy.ml/c/lemmy_support
• Did you check to see if this issue already exists?
• Is this only a single bug? Do not put multiple bugs in one issue.
• Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Summary

Similar to #3173, I'm having an issue where larger images fail to upload. Unlike most of the other related issue threads I've seen, though, my issue only appears when an image larger than a few hundred kilobytes is uploaded, as anything around a megabyte or lower uploads without issue. This is extremely odd, as I'm currently sitting at around 15% disk utilization. I tried some of the solutions mentioned in #3173, including restarting that container, checking for 0KB snap files in particular folders, and ensuring my config files were set up correctly (I'm using the Ansible install). I couldn't find anything out of the ordinary other than perhaps my docker container indicating I'm running asonix/pictrs:0.4.0-rc.7, which appears to be a pre-release version of pictrs. Aside from that, I'm unsure what could be causing this issue. Not being able to upload images of even a fairly modest size breaks many of the use cases I had in mind for Lemmy; I don't see why images larger than that would be an issue.

Steps to Reproduce

1. Upload an image several megabytes in size from any UI element that allows such an action.
2. Observe the SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data that appears as a result.
3. Notice how this error does not appear for very small images.

Technical Details

The full log file is 12MB in size, which seems too large to post here all at once. Here's the snippet which I believe is relevant, but I'm not really sure what to make of it. It appears similar errors keep occurring, so I suspect this is related to the problem expressed above.

�[33mlemmy_1     |�[0m �[2m2023-06-26T02:18:52.086407Z�[0m �[33m WARN�[0m �[2mlemmy_server::root_span_builder�[0m�[2m:�[0m Request error: error sending request for url (http://pictrs:8080/image/original/2bbfc82a-c9f0-4f54-89e2-172a1aa20c4c.png): error trying to connect: dns error: failed to lookup address information: Try again
�[33mlemmy_1     |�[0m Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("pictrs")), port: Some(8080), path: "/image/original/2bbfc82a-c9f0-4f54-89e2-172a1aa20c4c.png", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("dns error", Custom { kind: Uncategorized, error: "failed to lookup address information: Try again" })) })
�[33mlemmy_1     |�[0m �[2m2023-06-26T02:18:52.206083Z�[0m �[33m WARN�[0m �[2mlemmy_server::root_span_builder�[0m�[2m:�[0m Request error: error sending request for url (http://pictrs:8080/image/process.webp?src=9e59f02c-e85f-4425-88f1-57007407c83a.jpeg&thumbnail=256): error trying to connect: dns error: failed to lookup address information: Try again
�[33mlemmy_1     |�[0m Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("pictrs")), port: Some(8080), path: "/image/process.webp", query: Some("src=9e59f02c-e85f-4425-88f1-57007407c83a.jpeg&thumbnail=256"), fragment: None }, source: hyper::Error(Connect, ConnectError("dns error", Custom { kind: Uncategorized, error: "failed to lookup address information: Try again" })) })
�[33mlemmy_1     |�[0m �[2m2023-06-26T02:18:52.614757Z�[0m �[33m WARN�[0m �[2mlemmy_server::root_span_builder�[0m�[2m:�[0m Request error: error sending request for url (http://pictrs:8080/image/process.webp?src=c300d352-210d-4505-81a7-44475f60974b.jpeg&thumbnail=256): error trying to connect: dns error: failed to lookup address information: Try again
�[33mlemmy_1     |�[0m Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("pictrs")), port: Some(8080), path: "/image/process.webp", query: Some("src=c300d352-210d-4505-81a7-44475f60974b.jpeg&thumbnail=256"), fragment: None }, source: hyper::Error(Connect, ConnectError("dns error", Custom { kind: Uncategorized, error: "failed to lookup address information: Try again" })) })
�[33mlemmy_1     |�[0m �[2m2023-06-26T02:18:56.218313Z�[0m �[33m WARN�[0m �[2mlemmy_server::root_span_builder�[0m�[2m:�[0m Other errors which are not explicitly handled

Version

BE: 0.18.0

Lemmy Instance URL

https://normalcity.life

[AlphaHasher]

Can agree, I can't upload any images to anything

[delendum]

I suspect this may be docker related, I can upload large images fine on my (non-docker) instance.

[shreejalmaharjan-27]

+1

[0xAnansi]

I'm pretty sure the root cause it the same as the issue I've open here: LemmyNet/lemmy-ui#1532

Edit: you can set client_max_body at something like 5m in your nginx.conf to get a upper limit before it breaks

[EuphoricPenguin]

I'm pretty sure the root cause it the same as the issue I've open here: LemmyNet/lemmy-ui#1532

Edit: you can set client_max_body at something like 5m in your nginx.conf to get a upper limit before it breaks

This sounds exactly like the issue I'm having.

[EuphoricPenguin]

@0xAnansi Huh, my nginx_internal.conf has it set at 20 megabytes.

# Upload limit, relevant for pictrs
        client_max_body_size 20M;

EDIT: I see what you're saying. A lower limit than 20 is needed to ensure that the issue doesn't occur.

EDIT 2: I tried 5M, but it still throws errors for images that are much smaller than that, such as 2.5MB.

[0xAnansi]

@EuphoricPenguin you need to set the limit on the external nginx config too.

Here this is the external that rejects the request, and never sends it to the internal.

[EuphoricPenguin]

@EuphoricPenguin you need to set the limit on the external nginx config too.

Here this is the external that rejects the request, and never sends it to the internal.

Where is the external config file located? I couldn't see any more in the /srv/ directory.

[0xAnansi]

It's located in /etc/nginx/sites-available/

[EuphoricPenguin]

It's located in /etc/nginx/sites-available/

This one was already set at 10M, but I changed it and the internal one to 5M, and I'm still getting errors for smaller (2.5MB) images. Is this five megabits or five megabytes?

EDIT: Now, images of any size cannot be uploaded. I let the Ansible playbook reset everything, and now things are somehow worse. Even a 2KB image fails to upload now, but the images that are already hosted render fine.

[EuphoricPenguin]

lemmy_1     | 2023-06-28T16:13:12.934381Z  WARN lemmy_server::root_span_builder: Request error: error sending request for url (http://localhost:8080/image/process.webp?src=81b882ff-09c4-47e1-a2e8-e9dccd4d9628.png&thumbnail=96): error trying to connect: tcp connect error: Address not available (os error 99)
lemmy_1     | Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8080), path: "/image/process.webp", query: Some("src=81b882ff-09c4-47e1-a2e8-e9dccd4d9628.png&thumbnail=96"), fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 99, kind: AddrNotAvailable, message: "Address not available" })) })

Now I'm getting logs that look like this, and basically, all images are failing to upload. Any idea on what the issue might be?

[EuphoricPenguin]

I figured out how to actually correctly edit the config files now thanks to this, and I'm also aware of changes to how front-end error handling works. Based on all of this and the fact that image uploading now supports sizable images, I consider my issue fully resolved.