-
Notifications
You must be signed in to change notification settings - Fork 334
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Server error after updating to 0.18.0 #1530
Comments
Note: pictrs logs show no errors. When trying to access the image directly at https://SITE_URL_REDACTED/pictrs/image/a29da3fc-b6ce-4e59-82b0-1a9c94f8faed.webp, it works fine in a browser. The issue appears to be that lemmy-ui is trying to load the image from 127.0.1.1:443.
|
UPDATE: Issue "fixed" by putting the In OP's case, it's going to a local IP instead, but it's using the wrong port. In both our cases, the URL it uses for pictrs is not calculated correctly for whatever reason. The UI could fail to contact pictrs for any variety of reasons, but that shouldn't be fatal. It should just break images. Previous comment text: I get a very similar error but it's not quite the same:
Just like OP, I can go to the URL that error is logging and it pulls the image just fine. No browser cache shenanigans, the request is being fully completed. My server is now completely inoperable and I can't downgrade to 0.17.4 due to the database upgrades. Anything I can do to not lose my data? |
Put pictrs and lemmy-ui on the external network if you used the ansible script. That should have been updated now though. LemmyNet/lemmy-ansible#96 (comment) have a look here - try following the steps to remove the site icon via the database |
Nice! Good thing I backed up my DB. I’ll try changing the network settings and restoring my DB. That did not fix my instance. I had to manually set the site’s image icon to null in the database to get my site working again. There are still many issues in both lemmy-ui and lemmy be logs. |
short term fix is to remove your site’s image icon. db query: then at least your site will work again, sort of. It loads at least. I am not sure federation is working 100% though and the pictrs errors still persist in the logs. |
Thanks! I just left that there for posterity, I added an edit to the top of my comment about how I "fixed" this. It was caused due to a network error, so to fix it I just gave the container internet access, even though it otherwise wouldn't need it. That fixed things immediately. Still, network errors shouldn't be fatal. Especially not for an icon. |
For an instance set up with Ansible: |
I can confirm that this issue is still present in Lemmy-ui 0.18.1-rc.2. Every single time the docker images restart I have to run the commands to delete my site logo or the site will be unreachable. I think this may be partially because my external NGINX proxy is on a different different host. When Lemmy-ui resolves the IP for my domain it is still using the local IP and port 443. If believe the fix would be to use the port specified by the pictures server in the internal NGINX configuration. I'm not aware of how to do this. I will test if this can be accomplished by configuring the pictures server to listen on 443 in the internal configuration. |
Is your "external" NGINX proxy running on the same host server as Lemmy? |
I’m using the docker containers. No additional nginx running on the host. |
I think this is happening because there is not any error handling when the icon is fetched:
https://github.com/LemmyNet/lemmy-ui/blob/main/src/server/utils/fetch-icon-png.ts So if that call fails for any reason, it breaks the SSR render. I think this should be changed to return a broken image icon, or a blank image, instead of aborting the entire SSR process. Otherwise, the pictrs service going down for any reason causes the entire UI to crash like this. That goes for any other fetches for external data too. If it's not mission critical (and images aren't mission critical), the SSR would ideally continue the render. |
I found a more permanent fix. The error seems to be caused by having the host name of your server set to the same thing as your lemmy instance domain-name. The easy and more permanent fix is to change the host name of your server! |
@randommouse interesting. I’ll give that a try. |
@slash-zero "It's always DNS" |
@randommouse: OK, I updated to:
Then, I changed my host's server name to not match the domain name, and rebooted. Local images were still not working after a The final step was to revert the lemmy.hjson back to So, to clarify, what I did was:
|
Had the same problem when I upgraded to 0.18.0, this fixed it. |
can you write the instructions to change the hostname? |
|
You'll have to change it in I would still like to see this issue changed to track the lack of error handling here :p And probably the fact that lemmy-ui is trying to SSR by polling the real public hostname instead of the internal hostname. It didn't used to do that in |
@madeofstown I was getting the error on my server as well during its shakedown tests. The hostname is definitely not the same as the domain name. The behaviour I'm seeing:
My conclusion: the pass through to Pictrs that Lemmy is doing, is not working properly somehow and becomes borked after a restart. (in case that wasn't clear, my pictrs is not exposed to the internet directly and Lemmy is providing a /pictrs endpoint which is used to access pictrs as the developers intended) I will investigate a little more to see what else I can find out. |
Check that your lemmy.hjson has the new pictrs format instead of the old pictrs_url.
I had to update my config file to the new pictrs format as well because everytime i updated via the ansible script it was using the config file to revert the lemmy.hjson to the old pictrs_url format causing the Server Error
…On Thu, Jul 13, 2023 at 8:46 AM, Martijn ***@***.***(mailto:On Thu, Jul 13, 2023 at 8:46 AM, Martijn <<a href=)> wrote:
***@***.***(https://github.com/madeofstown) I was getting the error on my server as well during its shakedown tests. The hostname is definitely not the same as the domain name.
The behaviour I'm seeing:
- Start containers
- Add icon through UI
- Test everything to see it works fine
- Check DB, icon URL points correctly to exterior domain name
- Restart containers
- Site is borked
- Enter DB, check icon URL. Still fine
- Delete icon entry from DB
- Site works (no restart needed)
- Re-add icon through UI
- Verify site to see it still works. It does (still no restart)
My conclusion: the pass through to Pictrs that Lemmy is doing, is not working properly somehow and becomes borked after a restart.
(in case that wasn't clear, my pictrs is not exposed to the internet directly and Lemmy is providing a /pictrs endpoint which is used to access pictrs as the developers intended)
I will investigate a little more to see what else I can find out.
—
Reply to this email directly, [view it on GitHub](#1530 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/A3H6HEYQFB5XEYSR5KRTCVTXP7U2ZANCNFSM6AAAAAAZR66LVU).
You are receiving this because you commented.Message ID: ***@***.***>
|
@slrgt no, that's set correctly. It doesn't matter whether you do a whole The behavior remains the same. Lemmy suddenly tries to use the external IP and port 443. That causes the entire service to be unreachable due to lack of basic error handling in Lemmy UI. Simply removing the site icon URL from the DB corrects that behavior. The site then immediately becomes available again without the need to restart or anything. edit: just to be clear... re-adding the site icon doesn't break the site. The issue doesn't reappear until after the containernis restarted. |
In your docker config have you added the |
It's a bit silly to have the UI call out to the internet, just to come back to the same server on the same IP. This might even break routing in some situations. What should happen is, Lemmy-UI should have a config option to set the Pictrs URL, just like the Lemmy backend does. That way, in a Docker setup, it can connect directly to Pictrs via Docker internal routing. And in any other setup, it's configurable. |
I agree that this is a bug. I am just trying to help people work around it. When I said:
I meant that is how things work in the current implementation. |
Pictrs is not directly accessible through the internet. It is fronted by Traefik, which has no rules to forward traffic to pictrs. In other words: browser -> traefik -> lemmy -> pictrs Keep in mind that this setup works fine until a container restart. After the restart, simply removing and re-adding the site icon in the DB is enough to make it work again. (i.e. no need to restart again) This leads me to believe the issue is not DNS or routing based but something inside Lemmy itself. Putting pictrs on an external network temporarily "solves" the problem because it bypasses Lemmy. I have seen quite a number of weird issues during shakedown testing and it has led me to believe that Lemmy is not production ready in any sense of the word, so I'll probably be evaluating the situation with Kbin. |
This issue is back worse than before with 0.19. The previous solution of changing you server hostname no longer works and the database entry must be cleared manually. I cannot use lemmy-ui consistently but fortunately I've figured out how to run Photon on the base domain. |
Requirements
Summary
Docker setup. After updating to the 0.18.0 images for both lemmy-ui and lemmy backend, the lemmy-ui logs show a pictrs error, and the main site returns a "Server error".
Steps to Reproduce
Technical Details
Docker setup. After updating to the 0.18.0 images for both lemmy-ui and lemmy backend, the lemmy-ui logs show a pictrs error, and the main site returns a "Server error".
Lemmy Instance Version
0.18.0
Lemmy Instance URL
No response
The text was updated successfully, but these errors were encountered: