Focal point support for images

[lukas-schaetzle]

Summary

A simple tool for clicking on a Directus image asset and saving the X/Y pixel coordinates for a point of interest (focal point). This would be useful in many situations, especially dynamic thumbnail cropping.

Original idea by @benhaynes

Motivation & Basic Example

A user wants to display the following image with a different aspect ratio. However, the displayed section needs to be centered around the moon (green rectangle) and not around the center of the image (red rectangle).

Detailed Design

Sharp currently has no inbuilt support for defining a custom focal point, so we have to do the cropping manually. For a simple approach see my comment in the sharp repository: lovell/sharp#2740 (comment)

After the focal point for an image has been changed, all thumbnails need to be regenerated. We can do this by just deleting all generated thumbnails after the focal point change.

If there is no focal point configured, the thumbnail is generated around the center, just like it's the case now. (Basically the same as a centered focal point but we can just keep null stored in the database.)

UI:

Add a new "focal point" tool to the image editor. After selecting it, the mouse cursor changes to the cross shape and allows selecting the focal point by clicking into the image. Clicking outside of the image sets the focal point to the nearest valid place. The current focal point is indicated by a crosshair (maybe slowly blinking for better visibility) and is only shown if both x and y coordinates exist and are valid. The toolbar shows the pixel values and also allows manual input of the numbers. Trash icon for removing the focal point and AI icon for setting the focal point via an AI library/service (optional).

API:

The assets endpoint returns the focal point information as part of the image meta data.

focal_point: {
   original: {
      x: ...,
      y: ...
   },
   presets: {
      presetName1: {
         x: ...,
         y: ...
      },
      presetName2: {
         x: ...,
         y: ...
      }
   }
}

By default, the coordinates are pixel values. Optionally, we could also support requesting the coordinates as percentages (with query parameter). The preset focal point coordinates are generated on the fly when they are requested so they don't have to be saved to the database. (I'd expect the added latency to be very short.) Same for requesting the coordinates as percentages.

The SDK could provide a helper function to calculate the focal point coordinates for a dynamic transformation request:

type Coordinates = {
   x: number,
   y: number
}

function calculateThumbnailFocalPoint(
   original: {
      dimensions: Coordinates
      focalPoint: Coordinates
   },
   thumbnail: {
      dimensions: Coordinates
      fit: 'cover' | 'contain' | 'inside' | 'outside'
      withoutEnlargement: boolean
   }
): Coordinates

Database:

Table directus_files needs 2 new columns to store the focal point pixel coordinates (e.g. something like focal_point_x and focal_point_y). I chose pixel values because we will lose precision for big images if we choose to save percentages.

Requirements List

Must Have:

• Button in image editor which can be used for viewing/editing the focal point.
• Allow adjustment of the x and y coordinates via number input field
• Allow adjustment of the x and y coordinates by clicking in the image
• Focal point validation: it must be contained within the image dimensions (also when cropping the image)
• Image meta data has new focal point information
• Generate thumbnails around focal points so that the focal point is as close as possible to the center without having whitespace at any side
• Thumbnails need to be regenerated after changing the focal point

Should Have:

• Rotating, flipping and cropping the image in the image editor changes the focal point coordinates accordingly. (Check after cropping: If focal point is outside of image dimensions, set it to the nearest valid position.)

Could Have:

• Automatically set a focal point x, y with an upload hook to an AI service/library
• Image meta data could also include the shifted focal point coordinates for all asset presets
• SDK helper function which calculates the focal point coordinates for a generated thumbnail by providing thumbnail dimensions, original image dimensions and original focal point coordinates
• Get Coordinates as percentages instead of pixels

Won't Have:

• Ability to set multiple focal points per image or set it with the transformation api
• Ability to set where the focal point should be positioned in the generated thumbnail. (It will always be positioned as close as possible to the center.)
• Consider focal point for advanced transformations (passed arguments to sharp api)

Alternatives

Currently, you could provide focal point information via image tags, description or a custom field and use that to crop the image manually in the frontend. But then you can't use Directus' inbuilt thumbnail generation and it's more a hack like a real alternative.

Adoption Strategy

As a migration strategy, do not set a focal point for all existing images. That way, this would be no breaking change and exisiting thumbnails would not have to be regenerated if the focal point is not changed by the user.

[NilsBaumgartner1994]

the library smartcrop could be used for this

https://29a.ch/sandbox/2014/smartcrop/examples/testbed.html

[rijkvanzanten]

Heya! Thanks for opening this feature request!

This feature request has received over 15 votes from the community. This means we'll move this feature request to the Under Review state!

The Core team will schedule a meeting to review this request as soon as possible. The discussion will then be approved or denied. You may or may not be invited to join this meeting with the core team.

For more information, see our Feature Request Process.

[lukas-schaetzle]

I'm tempted to give this a try and would see the following things on the to-do list for this feature:

• The image editor should have another button which can be used for viewing/editing the focal point.
• It should allow manually adjusting the x and y coordinates...
• ...and setting the focal point by clicking in the image.
• By default the focal point is the center of the image
• Make sure that the focal point is contained within the image dimensions (also when cropping the image).
• Rotating and flipping the image should also change the focal point coordinates accordingly.
• Image meta data should have new focal point information:

focal_point {
   x: ...,
   y: ...
}

• Generate thumbnails around focal points so that the focal point is as close as possible to the center without having whitespace at any side
• Thumbnails need to be regenerated after changing the focal point

Open questions:

• How should we save the coordinates? As pixels or width/height percentages?
• When retrieving a transformed image, how can the user get the new possibly shifted focal point coordinates

[rijkvanzanten]

I find that behavior a bit unexpected and its often misunderstood. But since probably the absolute majority of users will use this anyway later in their CSS to align images to their focal point, we may get away with just doing it the same way in the backend.

Another interesting point. I imagined that this focal point would be used in the /assets endpoint to generate the thumbnail at the correct focal point instead of in CSS 🤔

[lukas-schaetzle]

Sorry for the late reply. Sure, we don't do it with CSS in the /assets endpoint. I only wanted to suggest that we could do it like the mentioned CSS attribute so that we don't have the issue of shifting focal point coordinates. So if people like me and bee8bit want to further crop the image in the front end via CSS (because of responsive and other dynamic behavior) we know where the focal point is in the crop we get from the /assets endpoint. But I'm still not sure if that's really better than another utility endpoint to get the coordinates...

If we save the focal point with percentages we also have to make sure that we choose a reasonable amount of saved decimal places for the focal point. I think we should be good with 2 (e.g. a saved coordinate would look like this: 30.55%). With that, we would still have an accuracy of 1px for image sizes up to 10,000px.

[lukas-schaetzle]

I put some thought in it again and came up with yet another option: The image meta data could also include the shifted focal point coordinates at least for all asset presets. So it would look like this:

focal_point: {
   original: {
      x: ...,
      y: ...
   },
   presets: {
      presetName1: {
         x: ...,
         y: ...
      },
      presetName2: {
         x: ...,
         y: ...
      }
   }
}

I assume that for all presets the focal point in the original image is the same. That being said, there may also be some folks who want to change the focal point as part of the transformations. But I don't think that would make much sense since I'd imagine that you can only position one focal point in the image editor...

So all in all, I think the probably best option would be the easiest one for now:

• there's only 1 focal point per image and the focal point is not part of the transformations
• always create a thumbnail with its focal point centered as much as possible
• Only provide the focal point coordinates for the original image. Optionally, the meta data could also contain the shifted coordinates for all asset presets. Otherwise, if a developer wants to know the new focal point coordinates for a thumbnail, they have to calculate them with the help of the thumbnail dimensions, original image dimensions and original focal point coordinates (SDK could provide a helper function for that)

[benhaynes]

Really great input! I would add that it could be a nice integration to automatically find a focal point x, y with an upload hook to an AI service.

[NilsBaumgartner1994]

Really great input! I would add that it could be a nice integration to automatically find a focal point x, y with an upload hook to an AI service.

Checkout Smartcrop (https://www.npmjs.com/package/smartcrop). Was using this in a real project for my clients which post daily around 10 images of food. Was working really nice.

[NilsBaumgartner1994]

That’s a good point.Beste Grüße,Nils BaumgartnerAm 01.07.2023 um 18:29 schrieb Lukas Schätzle ***@***.***>: Before implementing any automatic solution which the user can't control we should have a manual control. (Smart crop may not always identify the right crop the users wants.) After this is implemented, also adding an automatic solution (which can be overwritten by users) by default may be a good idea. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[NilsBaumgartner1994]

This is a very clever feature. Beste Grüße,Nils BaumgartnerAm 02.07.2023 um 10:57 schrieb Martin Melcher ***@***.***>: Some more elaborate suggestions, possibly for a later version: I've seen clever implementations in some CMSes which do not only store the focal point but also a resizable rectangle around it. The image is never cropped beyond this rectangle. Other aspect ratios try to keep the focal point in the same relative position. If the focal point is in the top left corner of the rectangle, the cropped image first grows to the bottom right, keeping the focal point in the top left corner. I've also seen the combination of a focal point and a vector in which the image should grow, but I found that both unintuitive for the users and less expressive (no minimal crop). —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[d3v1an7]

In the past, I've achieved what is being described using Cloudinary, which has a very extensive transformation API, and deals with this (and many other cropping requirements) through an attribute they named 'gravity'.

The API reference has a few mentions of how this could be implemented:
https://cloudinary.com/documentation/transformation_reference#g_special_position

custom - Use custom coordinates that were previously specified (e.g., as part of the image upload method) and make them the focus of the transformation. Defaults to center gravity if no custom coordinates have been specified.

I had also dabbled with imgix, which uses a attribute called Focal Point Crop.
https://docs.imgix.com/apis/rendering/focalpoint-crop

Cloudinary used X/Y values, which meant I needed to also know the original width/height of the image to do calculations.
imgix used percentages, which was much easier to implement and reason, but was 'lossy' when doing reallllly close in crops of large images (in hindsight, this was probably more to do with my math when converting values from the cropping tool than anything to do with imgix).

One thing I don't think I've seen discussed yet above is a relatively important part of the crop -- the 'zoom'.
The imgix focalpoint link demonstrates this concept well.

Imo, a friendly custom crop is flexible/chainable mix of the following:
focal point + zoom + aspect ratio of crop + width of crop

Having the existing Directus edit crop tool store those values against the image would be 👨‍🍳💋.

Side note: a much loved/loathed feature I built into the last newsroom media library I worked on was using a gravity of 'face detection' instead of needing to manually specific a focal point :) When it worked, it saved heaps of time, and people loved it -- but the early days also had some interesting results with sports photos, where the auto detected face would be some blurry face in the crowd 😂

[rijkvanzanten]

Imo, a friendly custom crop is flexible/chainable mix of the following: focal point + zoom + aspect ratio of crop + width of crop

Having the existing Directus edit crop tool store those values against the image would be 👨‍🍳💋.

This sounds pretty doable to me! We already have the editing interface to achieve the values for 3/4s of those, and it doesn't sound that crazy to add a focal point selection into the mix there 🤔

[rijkvanzanten]

I believe this has all the technical details in the comments above. I would love to shape this into the RFC template to make sure we can start going to implementation 🙂 (cc @w0kyj)

[lukas-schaetzle]

Edited my first post to bring it into the RFC template form.

[rijkvanzanten]

You da man 💪🏻

[rijkvanzanten]

@tummerhore The only thing I'm not entirely sure about is focal point returned from the file meta information based on the presets optionally as percentages. How / where would that be used? Isn't that something you can calculate out on the fly when it's needed (f.e. with an SDK helper)? I'm a tad worried about the implications of a generated field that has a different output from it's input 🤔

[lukas-schaetzle]

To paraphrase you (because I'm not 100% sure I understood you right): You're worried that the Directus API would return generated fields not saved in the database (shifted focal points for preset transformations and/or percentage instead of pixel values) because we could also just generate them on the client side (f.e. with a SDK helper), right?

But why exactly do you think that additional generated fields returned by the API are a concern? In comparison, I would argue that my proposal would greatly improve the developer experience without any downsides to it. Let's look at different possible applications when a developer requests a preset thumbnail (with key my_preset) and wants to know its focal point:

1. Directus API can return focal points for preset transformations directly (see RFC):
   The developer can simply request the desired data:

   query {
      focal_point {my_preset {x y}
   }
   

2. Directus REST API only returns original focal point. SDK has a helper function:
   The developer must request the original focal point but also the original image dimensions

   query {
      focal_point {x y}
      height
      width
   }
   

   Additionally, the developer must know the preset transformations on the client. So they add an object to the client where they duplicate the preset transformation details from directus. (DRY principle violated.):

   const presets = {
      my_preset: {
          height: <...> ,
          width: <...> ,
          fit: <...> ,
          withoutEnlargement: <...>
      }
   }

   Alternatively, they could also request that data from Directus. Either way, you would have to push additional data over the network, even though the developer may actually not need it besides from calculating the focal point.

   Finally, the developer must call the SDK function which then calculates the coordinates. The helper function could also allow inputting the preset key, so that it gets the preset transformations for you. But that would add latency, since it would not be called in parallel to the developer's first request.

   So all in all, we have more developer effort, more data transported over the network (original height and width + preset transformation details) and possibly more latency (synchronous call to get preset transformation details).

3. Directus REST API only returns original focal point. Developer does not use SDK:
   Currently, you can use Directus very easily without the SDK. However, if a developer not using the SDK wants to know a thumbnail focal point but that information is only provided by the SDK, they would have a problem since the focal point calculation is an implementation detail an external developer should not have to worry about. Of course, we could also introduce another rest endpoint for the calculation but calling that would also increase latency.

With concept no 1, a developer only has to use the SDK helper function when he wants dynamic thumbnails (transformation details passed as query parameters), which probably is only rarely the case for an application in production. (And then the transformations details are already known on the client anyway.)

[lukas-schaetzle]

If you're rather wondering about use cases why someone wants to know the focal point position in the generated thumbnail, I see 2 cases here:

1. The one I've already tried to explain before and which is quite common I guess: Your frontend displays the thumbnail in a responsive way so that the aspect ratio may change. Therefore, the focal point has to be considered also on the client. The easiest option here is to use CSS background-position with percentages. That's why it would be nice if we could also directly (and only explicitly via query parameter) request the focal point as percentage values.

   Example: A card component with a side image and a description. You don't know the length of the description so you don't know the aspect ratio of the final image. You don't want to just request the original image size, since content editors may upload extremly large images (already had the pleasure of learning about that 😥) or you may want to use different resolutions for different screen sizes.

   

2. If you want to position something relative to the focal point. (a circle, an arrow, ...) This may be not so common but I could see this also being wanted.

[rijkvanzanten]

But why exactly do you think that additional generated fields returned by the API are a concern? In comparison, I would argue that my proposal would greatly improve the developer experience without any downsides to it.

Good question! My concerns aren't really so much around the details of how this particular field would operate, or the value it adds, but that we currently don't have a system yet that would allow us to setup that type of generated field in the API, which in turn we'd effectively have to treat #3079 or #4231 or as a blocker for this feature to land, which is scary! I do agree that it'd be useful to have that output though. The reason I'm asking for the use case is to evaluate if it would be a requirement or more of a nice to have to be able ship the grunt of this feature standalone 🙂

[lukas-schaetzle]

I see. Didn't think that including the generated fields would be such a great deal in this case (especially because I wouldn't have added any caching to it for the beginning). But you can surely judge that better.

I wouldn't treat the generated fields as a requirement for the initial implementation since you could work around it with a SDK helper. (That's why I listed this feature in the "could have" section.) But adding it should definitely be long term goal.

[rijkvanzanten]

This feature request has been accepted and is queued to be implemented! You can follow along with the progress here: #20838