[Investigation] Image Processing & Manipulation

[ErisDS]

This issue replaces (at least for now) issues #1688, #1734, and #4333 as the one true place to discuss all of our image processing needs in Ghost, and the possible solutions.

Please note that this issue is not about storing or managing uploaded images, only about processing them on the way to the server ;)

---

Image Processing Features

When it comes to uploading images in Ghost, there are a few issues we need to be able to resolve:

• cropping an image to a square (see Add library to crop profile images when uploaded #4333 for full details)
• removing exif orientation data (see Image upload via iOS / Android displayed sideways #1688 for full details)
• simple 90° rotations (so that we get the right orientation of an image)
• optimising an image for display (resizing enormous digital camera photos to a sensible file and display size see Enhancement: Resize images to reasonable size when uploaded #1734)
• generating different sizes of an uploaded image for use in different situations (see Multiple image sizes below for more detail).

Multiple image sizes

There are a few ideas for Ghost features that have been floating around forever that would require us to be able to generate multiple sizes of any one image:

• serving different sized images to different devices, so that a blog is more optmised for mobile
• providing themes various image sizes to display in different places (i.e. featured image might be small on the index, and a huge cover image on the post page) which would be accessible by doing something like `{{image size="small"}}
• allowing themes or apps to define the image sizes they use (and Ghost generating them)
• any sort of image management app would need a thumbnail

The current state of Image Processing

Ghost doesn't currently have any form of image processing built in because we haven't yet found a good solution for doing this. Pretty much all node libraries for image processing have imagemagick as a dependency - and that is not a viable solution for Ghost due it it being a c++ program that's ridiculously hard to install. This leaves us with a few alternative options:

1. Client - Do the processing on the client side
2. PureJS - Find a pure JS module for image processing
3. Compiler - Compile imagemagick using something like emscripten
4. node-pre-gyp - Create or change an existing node module which wraps imagemagick to use node-pre-gyp which is the same thing we use to install sqlite3
5. API - Use a 3rd party API for image processing
6. Other Some other solution we haven't thought of yet.

Several of these solutions have been discussed or covered in the comments on #1688 and #1734, so they are worth a read ;)

What we know so far

1. Client - it may be possible to do simple cropping or resizing using the canvas element and/or File API, but this won't work for any sort of regeneration of sizes, so it won't present a full solution. JavaScript-Load-Image (mentioned here is probably worth looking into though).
2. PureJS - lwip is the only one I knew of, mentioned in Enhancement: Resize images to reasonable size when uploaded #1734, but it turns out even that is actually largely written in C/C++! Is there really not a single PureJS one out there?! This needs investigating.
3. Compilers - there was quite an extensive discussion on modules that we could potentially compile into JS and how on Image upload via iOS / Android displayed sideways #1688 (comment), this is worth further investigation as we might be a few small steps from a solution here.
4. node-pre-gyp - if we could identify one of the imagemagick wrapper projects which has the features we need and might be active and willing to accept a PR this might be a viable option. node-pre-gyp has taken pretty much all of the pain out of sqlite3 installs. Still, as this is nowhere near as important a dependency as sqlite, I wonder if it will resolve the issue quite enough to be worth the problems some people will still have.
5. API - there are a few that have been mentioned although I can't find the references now. Worth investigating, but this is likely to require keys and other complexities.
6. Other - anyone got any bright ideas?

---

In summary, there's quite a bit of research and investigation to be done to come up with a couple of potential solutions for image processing in Ghost, and weigh up their pros and cons. I don't think there is going to be an obvious winner, but we need to gather together a clear picture of what's available to us so we can make a decision. The key thing is that we only* need crop, rotate, resize, exif modification & some optimisation tools - we aren't trying to detect faces or anything!

Anyone and everyone is welcome to jump in on this, with any solution they can come up with - I'm not looking for one person to do all the research but rather for people to volunteer their suggestions.

* I know I know only... haha

[PaulAdamDavis]

Doing any form of image processing client-side is a definite no from my point. I've worked on a project before which required resizing an uploaded image in canvas, but the browser (iOS 7 Safari in this case) had a file size limit of approx 2mp. I couldn't even process photos from the devices own camera. :)

Processing images that the browser couple code with was also super slow.

Edit:

The maximum size for a canvas element is 3 megapixels for devices with less than 256 MB RAM and 5 megapixels for devices with greater or equal than 256 MB RAM. ~ Scroll to "Know iOS Resource Limits"

I know this is iOS specific, but it's a large audience.

[mattiascibien]

I use cloudinary for image processing on my blog and my (in development) node art gallery. The Node API is fantastic and works, the only problem is that by being a third party service and so could be better served as a plugin (or app in Ghost terms).

For using a pure nodejs library i guess that JIMP can be an option.

It can actually be useful for creating the Multiple image sizes @ErisDS mentioned.

[ErisDS]

@mattiascibien JIMP is an awesome find - thank you! It looks like it has most of the features we need - crop, resize and quality that will help us generate images.

This just leaves the exif device orientation issue, as I don't think it will solve that.

Would be great if someone could try JIMP out and see how well it works.

[mattiascibien]

@ErisDS I'll set up a simple repo on my profile and share my test with the Ghost team. Always wanting to learn more node. 👍

EDIT: for exif https://www.npmjs.org/package/exif-rotaten https://www.npmjs.org/package/fix-orientation and https://www.npmjs.org/package/jpegorientation. (I'll check those tomorrow since it is almost midnight here)

EDIT 2: Maybe with just a node exif parser and JIMP we can manipulate the image as we want.

[Izhaki]

While I understand the need for a "one true place" for a discussion on images, the features discussed here are momentous and will take ages to implement.

I wonder if an agile approach wouldn't be more appropriate here, so we can have quick time-to-market with image related issues? What's more, some requirements are far less common than others. In other words, I don't think other issues need to be closed due to this 'central discussion place'.

My particular case is that I user OmniGraffle to generate diagrams. I export them as PNGs, with may having 200px width. Ghost stretches these images, so pixilation appears. It makes little sense to add white margins to fit the blog width, as you don't want these margins when sharing these diagrams on social networks (like Pinterest).

All I need is a way to tell Ghost not to stretch diagrams. As far as implementation goes, this is nowhere on par with "providing themes various image sizes to display in different places" or many other ideas proposed here.

[ErisDS]

@Izhaki what you're describing is how the theme handles the images, not Ghost itself. If you don't want full-width images, you can use a different theme or modify the one you're using (I assume Casper).

This is intended as a place to gather and share knowledge about what solutions are available for processing and manipulating images in node. There may well be many smaller issues to implement solutions in future once we've decided on a direction - but it helps to keep all of the problems we have described in one location so that we can evaluate potential solutions.

[ekulabuhov]

Hey guys,

I decided to try and tackle the EXIF problem, so I grabbed jimp and built a small demo. It's running on node and is pure JS, so no 'hard to install' dependencies. Also, I'm thinking of turning it into express middleware. Any thoughts?

Demo: https://imageware.herokuapp.com/
Example images: https://github.com/recurser/exif-orientation-examples
Repo: https://github.com/ekulabuhov/imageware

[ErisDS]

Hey @ekulabuhov this looks awesome! The main thing with making it middleware is would it work with the busboy setup in Ghost or would we need to change that?

I'd love to see a PR adding your new middleware to Ghost! 😀

[ekulabuhov]

Thanks @ErisDS! The busboy setup shouldn't be a problem because I'm using multer at the moment, which is a simple wrapper around busboy. What bothers me is the performance of this solution. The implementation is way too slow with real photos. Decoding and encoding JPEGs takes a big chunk of time from what I see. Nodejs is not a great candidate for long blocking operations from what I gathered.

[ErisDS]

I've been thinking about this quite a lot, off and on, over the past couple of months. It's such a killer that we haven't got basic image processing available to us, and I think we're going to need to be super creative with a solution if we move forward.

I tested out @ekulabuhov's demo and whilst it's not fast, it wasn't terribly slow either, and the laws of tech say it should only get faster, right? So it might not be a complete loss.

In addition, from reading through the (very minimal) documentation for blueimp it seems they support some resizing in the client - but whether you still need a backend or whether it can be done using canvas in browsers that support it is not 100% clear.

What blueimp definitely can do, is show a preview of the image being uploaded. I think we could do something clever and combine this with server-side workers. We can show the image before it uploads, and then after it uploads kick off a task to do the necessary processing. The user doesn't need to wait for it to finish, they can carry on working with the preview version.

If none of that works, there are definitely 3rd party services we could offload the work to over an API, and that approach could work very well.

I think there's more investigation and experimentation to be done, but I believe there is a solution out there somewhere!

[marcfallows]

I'm very interested in this feature. jimp seems like a pretty good solution. A PureJS solution feels like the most appropriate direction for this project.

As long as the chosen library is wrapped in some API that represents the features Ghost needs (crop, resize, etc) then a more appropriate library could always be dropped in later.

Attempting one feature to see if jimp will do the trick could be a worthwhile effort. "Multiple image sizes" is a good candidate as it only requires resize, and could easily fall back to presenting the original image until the processing is complete (if the processing does indeed take a significant amount of time).

I'm eager to help out in any way possible to see this land in Ghost.

[ErisDS]

Yes I'd love to see a demo which used blueimp's preview to mask the upload process taking a while. I really think the solution is out there it just needs someone with enough time & interest to have a play around with it and see what they can come up with.

[nii236]

Jimp looks good and should do the trick. At the moment I'm having to locally batch resize photos taken from my camera, otherwise I'll be uploading 2mb per photo!

[PaulBGD]

I proposed a way we could optimize images at #5487

[JamesJosephFinn]

I would like to submit this new, free and open source api into consideration:
ResponsiveBreakpoints.com
via Cloudinary.
I use the web gui for other non-ghost work, and it is great.

[nii236]

I think it would be better to handle the image processing internally, on the Ghost server itself just in case the API dies on us one day.

[JamesJosephFinn]

@nii236 I've never actually gotten it running myself (that's a wee bit above my head at the moment) but the tool is open sourced on github

[rdetert]

@ekulabuhov Do you have any info on how to setup sharp as a middleware layer to resize based on image url parameters?

[ErisDS]

I have a demo here: ErisDS@fb0a0fb.

[lovell]

Prebuilt binaries are now available in sharp v0.20.0.

[zkanda]

Can't wait to have this feature built in, @ErisDS any change it would get included in the next release?

[venku122]

Hello, I am a happy ghost user and node programmer. The features defined in this issue would be awesome to have in the official ghost release. Is there a current roadmap of when this might be implemented? Is there a way I could look into contributing the functionality myself?

[mg5thave]

Any thought to accomplishing this with server side image resizing + caching? For instance: https://stumbles.id.au/nginx-dynamic-image-resizing-with-caching.html

If Ghost were able to determine that it is serving a thumbnail, preview, or full-sized image, it could simply pass the size in the URL, and have nginx process the image manipulation via a location proxy on the /content/images/ folder.

[tsia]

Interesting idea but that would require nginx in the first place. I don’t think such a feature should be implemented by requiring a specific webserver. Many people may be using apache or lighttpd or something else. Or they may be using a hoster without router access.

[do-io]

Has there been any updates or work being done towards this end? I do have people that are intrigued by using Ghost, but I know they are not really the type to resize images themselves. If you are looking for help, let me know.

Also, is there thoughts or options of a media browser situation?

[nii236]

I've been tracking this issue for years now and I'm surprised it hasn't been solved yet. In terms of client side resizing, I've used picajs with great success.

[kevinansfield]

It's very much on our radar and has been discussed internally a number of times but no quick-win has emerged so far. Unfortunately we don't have the resources to tackle it just yet, it's a very large project to implement well and in a way that works across the huge variety of Ghost install environments and use-cases.

You can add your vote/input to the related feature requests on our Ideas board that we'll be referencing when we do get to the design & implementation stage:

https://forum.ghost.org/t/automatic-thumbnailing-resizing-of-image-uploads/475
https://forum.ghost.org/t/media-library-manage-all-files-in-one-place/675

[farcaller]

no quick-win has emerged so far

#1734 was opened in 2013, that's about 5 years ago. I can understand that image processing isn't the scope of ghost and no one on the team wants to tackle this but then maybe you just need to phrase it in a way that doesn't create false expectations?

[do-io]

Put in my votes.

I have wanted to use Ghost as my main blog platform, but I have a problem with a PaaS that doesn't acknowledge the main focus for our Search Engine brethren - Well structured and a speedy website.

To that end, a minimum of resizing an image - or option at least to size to a size that can be turned on or off seems like a good starting point.

Personally, I can write these things myself, but if I do I am more likely to utilize a cloud function implementation.

From a starting point, perhaps start with a solution that can be used on Google (Pro) that supports your hosting solution adding a hook that extends into other services and grows with time.

It may not give everyone everything and serve everyone at once, but it is a starting point.

Also, thank you for creating this platform. I really do prefer using Markdown for writing posts.

[parkerproject]

Good points, the success of any platform is listening to its users because without the users there would be no product.

[JohnONolan]

This thread has become a rant and is no longer useful, so I'm restricting it to maintainers.

I'll summarise the answers to the questions which keep coming up:

Why don't you just do it already?
If it was that straightforward, we would. It isn't straightforward at all, and to date there's still no good solution.

Then howcome [other platform] manages to do it so easily?
Because it is either centralised, or written in PHP. Both of which make solving this very easy.

Omg I can't believe it has been 5 years and nobody has fixed this, what a joke
The way open source works is that people who care about an issue write code to solve it. If you care about this issue, please write some code to solve it.

I'm not going to write any code, I don't have time for that!
There's your answer to why this issue has been open 5 years.

So what now?
We'll work on it as soon as we can. Right now there are other things which are more important to us.

But: Anyone else is more than welcome to contribute to this if they want to see it happen sooner rather than later. Hannah has posted very detailed info above, and even a prototype implementation :)

This is also a great read: https://nolanlawson.com/2017/03/05/what-it-feels-like-to-be-an-open-source-maintainer/

[naz]

Opened an issue in sharp repository regarding image being transformed even though no compression parameters were specified - lovell/sharp#1360 . We have a workaround for it but would be nice to figure it out 🤔

[allouis]

Cropping example here: https://github.com/tryghost/cropper

[kirrg001]

FYI: We have merged and released (Ghost 2.1.0) the first iteration of image processing using sharp. sharp is very good maintained, offers all the functionality we need and is easy to install since 1.20.0.