Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

502 on image upload, but works fine with ghost run #11129

Closed
ekerstein opened this issue Sep 16, 2019 · 126 comments
Closed

502 on image upload, but works fine with ghost run #11129

ekerstein opened this issue Sep 16, 2019 · 126 comments
Labels
bug [triage] something behaving unexpectedly

Comments

@ekerstein
Copy link
Contributor

I'm having a similar issue as #3827 and the post on the forum here.

I just updated to 2.31.1 and now whenever I upload an image I get a 502 error. I'm using a DO droplet.

I tried debugging with ghost run but then it works. However, when I go back to ghost start it gives me 502s again.

Technical details:

  • Ghost Version: 2.31.1
  • Node Version: 10.16.3
@ekerstein
Copy link
Contributor Author

Update:
I rolled back my ghost version to 2.30.2 and it started working again.

@andreborud
Copy link
Contributor

As you got it working again by downgrading I'm not sure if nginx is the problem. On my droplet running CentOS I had to change the max size of any file being uploaded in the base config of nginx located in /etc/nginx/nginx.conf. I don't remember the default size 2 or 5mb.

http {
    ...
    client_max_body_size 50M; <- change this line
    ...
}

@naz naz assigned naz and unassigned naz Sep 23, 2019
@naz naz added the support request [triage] A support request that belongs on the forum https://forum.ghost.org/c/help label Sep 23, 2019
@label-actions
Copy link

label-actions bot commented Sep 23, 2019

Hey @ekerstein 👋 We ask that you please do not use GitHub for help or support 😄. We use GitHub solely for bug-tracking and on-going feature development.

Many questions can be answered by reviewing our documentation. If you can't find an answer then our forum is a great place to get community support, plus it helps create a central location for searching problems/solutions.

FYI: Many projects have their own support guidelines and GitHub will highlight them for you as it did here, or the project owners will use issue templates to point you in the right direction. Reading the guidelines or issue templates before opening issues can save you and project maintainers valuable time.

@label-actions label-actions bot closed this as completed Sep 23, 2019
@ekerstein
Copy link
Contributor Author

@andreborud What do you recommend changing the client_max_body_size to?

@andreborud
Copy link
Contributor

Up to you, depends on how large files you plan to upload. I set it to 50 so I don't need to think about it too much.

@ivRodriguezCA
Copy link

ivRodriguezCA commented Oct 2, 2019

Dunno if it was the same problem. But in my case, the issue was that some images were stored in /content/images/{year}/{month}/ but referenced at content/images/size/w{number}/{year}/{month}/ when a GET request to size/w{number}/... came in Ghost would crash and restart. I had to manually copy and move some of them around.

For example:

I had to copy content/images/2019/06/hero-image.jpg into content/images/size/w2000/2019/06/hero-image.jpg for it to be loaded on the post and avoid Ghost crashing.

Edit: Adding this comment because editing /etc/nginx/nginx.conf didn't help with my problem.

@andreborud
Copy link
Contributor

@ekerstein Did you manage to get it working? I have run in to the same issue now. I haven't tried to upload an image for a while, tried last night and I get a 502 error.

@andreborud
Copy link
Contributor

andreborud commented Oct 7, 2019

In ghost I get "Something went wrong :(" and when inspecting I see this error: POST https://andreborud.com/ghost/api/canary/admin/images/upload/ 502

But going in to the error logs of ghost there is no 502 errors logged, the only persistent error is the following:

{
	"name": "Log",
	"hostname": "host",
	"pid": 4586,
	"level": 50,
	"req": {
		"meta": {
			"requestId": "1e7a7421-62f8-4ad2-9a32-a24020798489",
			"userId": null
		},
		"url": "/users/me/?include=roles",
		"method": "GET",
		"originalUrl": "/ghost/api/canary/admin/users/me/?include=roles",
		"params": {},
		"headers": {
			"host": "andreborud.com",
			"x-real-ip": "192.145.124.124",
			"x-forwarded-for": "192.145.124.124",
			"x-forwarded-proto": "https",
			"connection": "close",
			"sec-fetch-mode": "cors",
			"dnt": "1",
			"app-pragma": "no-cache",
			"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36",
			"content-type": "application/json; charset=UTF-8",
			"accept": "application/json, text/javascript, */*; q=0.01",
			"x-requested-with": "XMLHttpRequest",
			"x-ghost-version": "2.31",
			"sec-fetch-site": "same-origin",
			"referer": "https://andreborud.com/ghost/",
			"accept-encoding": "gzip, deflate, br",
			"accept-language": "en-GB,en-US;q=0.9,en;q=0.8",
			"cookie": "**REDACTED**"
		},
		"query": {
			"include": "roles"
		}
	},
	"res": {
		"_headers": {
			"x-powered-by": "Express",
			"cache-control": "no-cache, private, no-store, must-revalidate, max-stale=0, post-check=0, pre-check=0",
			"content-type": "application/json; charset=utf-8",
			"content-length": "321",
			"etag": "W/\"141-MpHclrM9KZiRaDmHfy2VdplNIzo\"",
			"vary": "Accept-Encoding"
		},
		"statusCode": 403,
		"responseTime": "9ms"
	},
	"err": {
		"id": "a7a38f30-e8cb-11e9-91b9-f398e47b2304",
		"domain": "https://andreborud.com",
		"code": null,
		"name": "NoPermissionError",
		"statusCode": 403,
		"level": "normal",
		"message": "Authorization failed",
		"context": "\"Unable to determine the authenticated user or integration. Check that cookies are being passed through if using session authentication.\"",
		"stack": "NoPermissionError: Authorization failed\n    at new NoPermissionError (/home/user/andreborud.com/versions/2.31.1/node_modules/ghost-ignition/lib/errors/index.js:114:23)\n    at authorizeAdminApi (/home/user/andreborud.com/versions/2.31.1/core/server/services/auth/authorize.js:76:25)\n    at Layer.handle [as handle_request] (/home/user/andreborud.com/versions/2.31.1/node_modules/express/lib/router/layer.js:95:5)\n    at next (/home/user/andreborud.com/versions/2.31.1/node_modules/express/lib/router/route.js:137:13)\n    at /home/user/andreborud.com/versions/2.31.1/core/server/services/auth/session/middleware.js:119:20\n    at /home/user/andreborud.com/versions/2.31.1/node_modules/express-session/index.js:495:7\n    at Child.SessionModel.findOne.then (/home/user/andreborud.com/versions/2.31.1/core/server/services/auth/session/store.js:24:28)\n    at Child.tryCatcher (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/promise.js:517:31)\n    at Promise._settlePromise (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/promise.js:574:18)\n    at Promise._settlePromise0 (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/promise.js:619:10)\n    at Promise._settlePromises (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/promise.js:699:18)\n    at _drainQueueStep (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/async.js:138:12)\n    at _drainQueue (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/async.js:131:9)\n    at Async._drainQueues (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/async.js:147:5)\n    at Immediate.Async.drainQueues [as _onImmediate] (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/async.js:17:14)"
	},
	"msg": "Authorization failed",
	"time": "2019-10-07T06:28:20.007Z",
	"v": 0
}

The NGINX error logs reports this:

upstream prematurely closed connection while reading response header from upstream, client: 192.145.124.124, server: andreborud.com, request: "POST /ghost/api/canary/admin/images/upload/ HTTP/2.0", upstream: "http://127.0.0.1:2371/ghost/api/canary/admin/images/upload/", host: "andreborud.com", referrer: "https://andreborud.com/ghost/"

I have also tried to temporarily disable SELinux, no difference.

@ivRodriguezCA
Copy link

Actually no. Yesterday I was publishing an article and had to ghost stop and ghost run, upload the image, then CTL + C and ghost start. Then manually copy the image from content/images/2019/10 to content/images/size/w2000 for it to work.

@andreborud
Copy link
Contributor

Stopping ghost and then start it with run and then start works for me as well.

@ekerstein
Copy link
Contributor Author

ekerstein commented Oct 8, 2019

@andreborud Honestly, I haven't tried against since rolling back to 2.30.2. So I just tested it again now:

  • Checked my nginx configuration and it was already at 50M. Also, the file I uploaded when I got a 502 error was only 1-2mb. So unless the config file is in another folder, I'm not sure that's the issue.
  • The nginx config also doesn't explain why rolling back to a previous version would make it work again.

I came across this issue related to the config server.port being 2369 while nginx was on 2368. I realized I had the same issue! So I did ghost config set server.port 2368, restarted, and updated to the latest version. My site loaded fine but when I went to try and access my posts it gave me a 500 error. I did ghost run and it gave me this:

InternalServerError: Cannot read property '0' of undefined
    at new GhostError (/var/www/ghost/versions/2.32.0/core/server/lib/common/errors.js:10:26)
    at _private.prepareError (/var/www/ghost/versions/2.32.0/core/server/web/shared/middlewares/error-handler.js:51:19)
    at Layer.handle_error (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/layer.js:71:5)
    at trim_prefix (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:315:13)
    at /var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:335:12)
    at next (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:275:10)
    at Layer.handle_error (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/layer.js:67:12)
    at trim_prefix (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:315:13)
    at /var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:335:12)
    at next (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:275:10)
    at /var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:635:15
    at next (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:260:14)
    at next (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/route.js:127:14)
    at apiImpl.then.then.catch (/var/www/ghost/versions/2.32.0/core/server/api/shared/http.js:97:17)
    at tryCatcher (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:725:18)
    at _drainQueueStep (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:93:12)

TypeError: Cannot read property '0' of undefined
    at forEach (/var/www/ghost/versions/2.32.0/node_modules/@tryghost/url-utils/lib/utils/mobiledoc-relative-to-absolute.js:22:43)
    at Array.forEach (<anonymous>)
    at Object.mobiledocRelativeToAbsolute (/var/www/ghost/versions/2.32.0/node_modules/@tryghost/url-utils/lib/utils/mobiledoc-relativ                        e-to-absolute.js:21:39)
    at UrlUtils.mobiledocRelativeToAbsolute (/var/www/ghost/versions/2.32.0/node_modules/@tryghost/url-utils/lib/index.js:387:22)
    at Object.forPost (/var/www/ghost/versions/2.32.0/core/server/api/canary/utils/serializers/output/utils/url.js:34:36)
    at Object.mapPost (/var/www/ghost/versions/2.32.0/core/server/api/canary/utils/serializers/output/utils/mapper.js:35:9)
    at frame.response.posts.models.data.map.model (/var/www/ghost/versions/2.32.0/core/server/api/canary/utils/serializers/output/post                        s.js:15:56)
    at Array.map (<anonymous>)
    at Object.all (/var/www/ghost/versions/2.32.0/core/server/api/canary/utils/serializers/output/posts.js:15:36)
    at serializeOptionsShared (/var/www/ghost/versions/2.32.0/core/server/api/shared/serializers/handle.js:107:58)
    at /var/www/ghost/versions/2.32.0/core/server/lib/promise/sequence.js:10:31
    at tryCatcher (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/util.js:16:23)
    at Object.gotValue (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/reduce.js:168:18)
    at Object.gotAccum (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/reduce.js:155:25)
    at Object.tryCatcher (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromiseCtx (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:641:10)
    at _drainQueueStep (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:97:12)
    at _drainQueue (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:15:14)

I rolled back to 2.30.2 again and everything is working. So now I'm not sure if the issue is related to the port 2368 that I changed or something else.

@ekerstein
Copy link
Contributor Author

FYI - v.2.34.0 fixed my 500 error issue.

However, now my 502 error on image upload (not occuring during ghost run) is back :(

@ErisDS ErisDS reopened this Oct 10, 2019
@ErisDS
Copy link
Member

ErisDS commented Oct 10, 2019

I have a feeling this is to do with sharp. I think maybe it doesn't always install properly, and when it doesn't, we're not falling back gracefully like we should.

In the case of a "recommended stack" setup, it should always install. @ekerstein can you look through your versions/xx/node_modules folders and see if sharp is in some or all of them?

We are looking into this, but it's slow work & very hard to track down because we haven't reproduced it yet.

Also can you run ghost doctor?

@andreborud
Copy link
Contributor

Updated to 2.34.0 this morning and the problem persists.

And sharp is installed on this and previous versions.

@ErisDS
Copy link
Member

ErisDS commented Oct 10, 2019

What does ghost doctor say?

@andreborud
Copy link
Contributor

Well I'm running ghost on CentOS so the doctor isn't to excited about that. You can see the output below.

$ ghost doctor
✔ Checking system Node.js version
✔ Checking logged in user
✔ Ensuring user is not logged in as ghost user
✔ Checking if logged in user is directory owner
✔ Checking current folder permissions
System checks failed with message: 'Linux version is not Ubuntu 16 or 18'
Some features of Ghost-CLI may not work without additional configuration.
For local installs we recommend using `ghost install local` instead.
? Continue anyway? Yes
System stack check skipped
ℹ Checking operating system compatibility [skipped]
✔ Checking for a MySQL installation
+ sudo systemctl is-active ghost_andreborud
? Sudo Password [hidden]
Instance is currently running
ℹ Validating config [skipped]
✔ Checking folder permissions
✔ Checking file permissions
✔ Checking content folder ownership
✔ Checking memory availability

However, I have been running ghost for a long time on CentOS without this problem, it popped up recently, not entirely certain what version introduced the bug. And stopping ghost and running it with ghost run instead make the image upload work again.

Is there any other information I can provide for you guys to make things easier?

@ekerstein
Copy link
Contributor Author

@ErisDS Checked some of my versions folders. The version I kept rolling back to that worked (2.30.2) has sharp version 0.23.0. The latest version of Ghost has 0.23.1. Not sure if that means anything.. but I'm happy to provide more details if I can help.

@timskap
Copy link

timskap commented Oct 18, 2019

I've got the same problem and image uploading works only with "ghost run". When I try use "ghost start" i've got error 502.

@ErisDS ErisDS added bug [triage] something behaving unexpectedly and removed support request [triage] A support request that belongs on the forum https://forum.ghost.org/c/help labels Oct 19, 2019
@andreborud
Copy link
Contributor

This problem still happens in Ghost 3.0.0

@shanteshpatil
Copy link

shanteshpatil commented Oct 23, 2019

This problem still happens in Ghost 3.0.0

I was really hoping this would be fixed in such a major version update. I am facing the same problem

Ghost CLI Version is 1.12.0

Hosted on a Digital Ocean droplet

@ekerstein
Copy link
Contributor Author

I tried going into the current folder and downgrading sharp to 0.23.0 just out of curiosity (since that was the version on an earlier working ghost install). It was messy because some of the dependencies got messed up, but I was able to get the server started again. No luck though, same issue persisted.

@ekerstein
Copy link
Contributor Author

ekerstein commented Oct 26, 2019

Maybe discovered something? I decided to just add logging extensively throughout the image processing files.

In lib/image/manipulator.js I added logging to everything. Then I uploaded a file to see the normal process vs broken process.

Based on the logging, this is my understanding of what happens when you upload a file. This is using ghost run and works fine.

  1. makeSafe
  2. makeSafe require('sharp')
  3. unsafeProcess
  4. unsafeResizeImage
  5. unsafeProcess return fs.writeFile

So then I tried it in production with ghost start. The server restarted in the makeSafe function at the same point every time. Right here:

const makeSafe = fn => (...args) => {
    common.logging.info('log 1');
    try {
        common.logging.info('log 2');
        require('sharp');
    } 
    ...

Every time it fails, "log 1" is logged, and then it never makes it to "log 2". It restarts immediately after "log 1" like this:

[2019-10-26 17:56:14] INFO log 1
[2019-10-26 17:56:20] INFO Ghost is running in production...
[2019-10-26 17:56:20] INFO Your site is now available on https://example.com/
[2019-10-26 17:56:20] INFO Ctrl+C to shut down
[2019-10-26 17:56:20] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED
[2019-10-26 17:56:20] WARN Tries: 0
[2019-10-26 17:56:20] WARN Retrying...
[2019-10-26 17:56:20] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED
[2019-10-26 17:56:20] WARN Tries: 1
[2019-10-26 17:56:20] WARN Retrying...
[2019-10-26 17:56:20] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED
[2019-10-26 17:56:20] WARN Tries: 2
[2019-10-26 17:56:20] WARN Retrying...
[2019-10-26 17:56:20] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED
[2019-10-26 17:56:20] INFO Ghost boot 4.203s

I assume it was failing on require('sharp'). So I removed that line and it continued on to unsafeProcess and then unsafeResizeImage where it failed again on this line where it's loading sharp again.

const unsafeResizeImage = (originalBuffer, { width, height } = {}) => {
    const sharp = require('sharp');

Seeing as the issue is sharp, I tried reinstalling Ghost, downgrading sharp to 0.23.0, and 0.22.1, but no luck.

@1c7
Copy link

1c7 commented Oct 28, 2019

Ghost 3.0

image
Unable to upload image. version 2.3 work fine. I just update Ghost to 3.0 yesterday and this happen.

I didn't post log because I am not sure which log I should look at.
let me know which log I should copy&paste here~

@kalinof
Copy link

kalinof commented Jan 30, 2020

Running ghost 3.4.0 on AWS EC2. I had the problem since ghost ver 2. Today I checked my nodeJs version. It was version 8. After updating to the latest version 10 everything seems to work again.

@brettdewoody
Copy link

Getting the same error on Ghost v3.0.1 and Node v10.13.0. 502 response from /ghost/api/v3/admin/images/upload/ and the logs show ghost is restarting when I try to upload an image.

From the above conversation it looks like installing libvips@8.7.4 and updating sharp to use that version is one solution. There's an open PR for updating sharp from 0.23 > 0.24, but no mention in the PR about this issue. The changelog for the new sharp version (0.24) has this commit:

Upgrade to libvips v8.9.0-rc4, drop support for Node.js 8

but I'm unsure if that addresses the issue we're seeing here.

Happy to help debug in any way.

@tonilampela
Copy link

I came across this issue related to the config server.port being 2369 while nginx was on 2368. I realized I had the same issue! So I did ghost config set server.port 2368, restarted, and updated to the latest version.

Oh my god... I've been struggling with the 502 error with many recent 3.x updates and had to always roll back. This was my exact problem too. Something is messed up with the 3.x upgrades regarding nginx port setup.

@elyseholladay
Copy link

Update:

I created a new DO droplet (for unrelated reasons, moving my site to a subdomain) with:
Ubuntu 18.04 Ghost Droplet
Ghost-CLI version: 1.13.1
Ghost version: 3.5.1
NodeJS: 10.16.3

I ran into an issue where my permissions on my content folder was incorrect, but after I fixed that, I can upload files just fine, without the libvips/sharp package version fix. 🤷‍♀ If creating a new instance is doable for you, it seems to be fixed now.

@LanceMaverick
Copy link

I've tried every fix here, including downgrading libvips version used by sharp and installing a brand new ghost instance and copying over content and although it solves the 502 error on upload, responsive images are no longer generated. Respective folders and images are not created inside content/images/size/. Has anyone managed to fix this part of the issue?

@darrenmhill
Copy link

This is a major issue! I'm shocked that it has still not been fixed given the number of Ghost releases. Is this bug not considered high priority?

@vikaspotluri123
Copy link
Member

@darrenmhill regardless of the importance of this issue, it's quite difficult to fix as we're not able to create a RCA with the updated version of libvps. In the meantime, many people have mentioned that migrating to a new droplet or downgrading libvps and upgrading sharp in your installation is a valid workaround.

@nickad
Copy link

nickad commented Feb 20, 2020

I got the same image upload problem described above when I installed a fresh copy of Ghost today on CentOS 7 from the zip file: https://ghost.org/zip/ghost-latest.zip

The relevant versions were:

Ghost-CLI version: 1.13.1
Ghost version: 3.7.0
Node 12.14.1

I had been running Ghost via systemctl so, after reading the above posts, I tried running it via "ghost run". It started ok but when I tried to upload an image I got this error:

Something went wrong installing the "sharp" module

/lib64/libz.so.1: version ZLIB_1.2.9 not found (required by /my/path/versions/3.7.0/node_modules/sharp/build/Release/../../vendor/lib/libpng16.so.16)

The advice was to:

I noticed my CentOS 7 distribution is running zlib 1.2.7.
I removed the /my/path/versions/3.7.0/node_modules/sharp directory then reinstalled sharp via:

npm install sharp

Then I started Ghost via systemctl and the image upload worked fine.

Just thought I would share that in case it is of any help in resolving this.

@BeanBagKing
Copy link

BeanBagKing commented Feb 22, 2020

Edit: It looks like it did work. However, I had to specify sharp v0.24, e.g. npm install sharp@0.24. The default npm install sharp installed v0.23.4

Having the same issue. Image uploads work fine under "ghost run", but fail when ghost is started normally.

I tried removing sharp from my current version (3.7.0), and then reinstalling via npm install sharp per the above instructions, but no luck. I actually had a few hours of screwing things up, force re-installing the current version, having dependency issues, fixing things, breaking them again. I can't begin to say exactly what I did, but none of it worked anyway.

Running Ubuntu 18.04.4 LTS in AWS. I'll be happy to provide any other information that might be useful, or try anything else that someone wants me to for troubleshooting. Just let me know what commands to run.

@tud0r
Copy link

tud0r commented Mar 1, 2020

This thread has been incredibly helpful thanks all who have contributed to the investigation of this! My set up too is a pre-installed Ghost set up on a DigitalOcean box. After reading through all this I tried the installation of a new ghost instance which ended up working.

I ran ghost install in a new ghost directory and backed up the old one.
The new installation required inputting details for a new mysql database, user, ssl etc etc. In order to get the new installation to work I had to go back and reconfigure the .conf file and map everything to my old database and also make sure all the permission were correct as mentioned in previous comments in this thread, before it finally worked.

@daniellockyer
Copy link
Member

Hi all! Just to let you know that Ghost 3.9.0 has shipped with sharp 0.24.1. You might need to update with --force if you run into trouble with the binary dependencies.

@colroyce
Copy link

colroyce commented Mar 3, 2020

@daniellockyer Thank you! I had this 502 issue and avoided upgrading Ghost in case it breaks temporary solution that’s described here. I upgraded Ghost (it didn’t require --force) a few minutes ago, and it works now. Seems like image upload problem is no longer with us.

@darrenmhill
Copy link

Excellent news! Confirmed working fine for me now. I'd missed this being closed and didn't see anything in the main release notes, but very happy my problems are now solved. Great job all.

@ErisDS
Copy link
Member

ErisDS commented Mar 5, 2020

We weren't 100% sure the issue was fixed - it's been impossible to reproduce, but have updated the release notes now.

@darrenmhill
Copy link

It's fixed for me and I and my team massively appreciate it! Thank you :)

@karai17
Copy link

karai17 commented May 12, 2020

I'm getting this issue using the ghost:latest docker image as of a couple days ago. Is there no plans to actually push a real fix for this, or are users expected to manually upgrade and downgrade various libraries themselves?

@pascalandy
Copy link

I confirm this was fixed as well. Thanks!

@adikari
Copy link

adikari commented May 9, 2021

I have just upgraded to ghost version 4 and having this same issue. When I downgrade to 3 I can confirm its working. I am running ghost in a docker container. I am suspecting, it could be a regression on ghost 4.

@nolo14
Copy link

nolo14 commented May 9, 2021

I have some problems with ghost upload images after upgrade a couples of days ago.

Ghost version: 4.4.0
Node version: 14.16.1
Ghost-cli version: 1.17.0
Server: ubuntu 16.04
First at all, when I update ghost it crash when so many error and famous message “We’ll be right back” after this I make a fresh ghost install and import my /content/images folder for new one. I reinstall node, ghost-cli, nvm, …

Now my blog works perfectly but I can’t upload images anyway in may posts… ghost error log show:

ERROR “GET /ghost/api/canary/admin/notifications/” 500 223ms
NAME: InternalServerError
MESSAGE: allNotifications.forEach is not a function

My nginx error log show:
2021/05/06 14:17:23 [error] 30072#30072: *4997 upstream prematurely closed connection while reading response header from upstream, client: X, server: blog.manuelfnavas.es, request: "POST /ghost/api/canary/admin/imag$

2021/05/06 14:17:52 [error] 30072#30072: *4997 connect() failed (111: Connection refused) while connecting to upstream, client: X, server: X, request: "POST /ghost/api/canary/admin/images/upload/ $

2021/05/06 14:35:34 [error] 30072#30072: *5086 upstream prematurely closed connection while reading response header from upstream, client: X, server: X, request: "POST /ghost/api/canary/admin/imag$

2021/05/06 14:35:38 [error] 30072#30072: *5086 connect() failed (111: Connection refused) while connecting to upstream, client:X, server: X, request: "GET /ghost/api/canary/admin/posts/?limit=30&$

I get working upload images if I disable Image optimization in config.production.json adding

“imageOptimization”: {
“resize”: false
}

Disabled image optimization works for post image but not for images in post body which follow crashing site 😦

trying with removed the /my/path/versions/4.4.0/node_modules/sharp directory then reinstalled sharp via:

npm install sharp

Not working…

Uninstall ghost, delete directory, create new directory and install ghost from 0 and after that import /content/images… same problem 😱

@pascalandy
Copy link

Try to reinstall using Node 12.

@nolo14
Copy link

nolo14 commented May 10, 2021

In a fresh install @pascalandy or only with ghost update --force after node 12 installation???

@pascalandy
Copy link

I can't help on that as I run things in Docker. But I'm sure this will help: https://forum.ghost.org/t/minimum-node-js-versions-changing-to-12-22-1-14-16-1-node-v10-is-eol/22527

@nolo14
Copy link

nolo14 commented May 10, 2021

Thanks so much @pascalandy but unfortunately downgrade to node 12 not working. Still same issue, upload photo crash and restart site :(

@OpenPrunus
Copy link

OpenPrunus commented May 11, 2021

Hello,

I have the same problem
Ghost 4.4.5
Node 14.16.1

And in nginx error log :

[error] 16768#16768: *61 upstream prematurely closed connection while reading response header from upstream, client: 192.168.1.254, server: www.ecuri.es, request: "POST /ghost/api/canary/admin/images/upload/ HTTP/2.0", upstream: "http://127.0.0.1:2368/ghost/api/canary/admin/images/upload/", host: "www.ecuri.es", referrer: "https://www.ecuri.es/ghost/"

I rollbacked to v4.3.3 with Node 12.20.1

:(

@adikari
Copy link

adikari commented May 12, 2021

Rolling back to 4.3.3 Alpine container fixed it for me. Must be something to do with the latest version

@nolo14
Copy link

nolo14 commented Aug 8, 2021

Anyone have a solution? I update to 4.11 still same issue. Image upload crash site :(

@AkkuratSoftware
Copy link

Ghost 5.4.1
Ubuntu 20.04

Was receiving the same 502 Error.

Resolved it in the following way.

cd var/www/blog/versions/5.4.1/node_modules
rm -R sharp
npm install sharp@0.24 --force <--force flag is needed as there is are dependency issues>

Did a ghost start and problem is resolved re: Image uploads. I will edit this post if I encounter any additional issues with the forced npm install.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug [triage] something behaving unexpectedly
Projects
None yet
Development

No branches or pull requests