Recent regression in display of image embeds without pre-existing dimensions data

[eramdam]

Steps to Reproduce

Taking this post as an example

https://bsky.app/profile/macthemes.bsky.social/post/3lbxkfazqdx2f

It showed "as expected" on 1.94

But shows up with objectFit: cover on 1.95, which obstructs a bunch of the image

I haven't checked if the issue happens on mobile or not but I woud assume so?

It seems like the regression was caused by the fallback square aspect ratio introduced in #6474

Which makes sense! Layout shifts suck, and it's an understandable fix, I just feel that retroactively changing the way images are displayed isn't great either?
My bot has been posting images for way longer than the aspectRatio field has been available on app.bsky.embed.images records so it feels "unfair" 😛

I fixed my posting code so it won't happen for future records, but I wonder if a middle ground "fix" would be to:

1. assume the aspect ratio is square
2. use the fetched dimensions to resize the image inside the square aspect ratio?

It might cause a slight flash/jump, but it's better than a layout shift and more importantly, wouldn't retroactively "hide" content for images that lacked aspect ratio data and displayed fine until 1.95

Attachments

No response

What platform(s) does this occur on?

Web (Desktop), Web (Mobile)

Device Info

No response

What version of the app are you using?

1.95

Additional Information

No response

[12Me21]

It wouldn't cause any flash/jump at all, just need to set css object-fit to contain (instead of cover), so all images with an incorrect/unknown aspect ratio will be scaled down instead of cropped

[GlitchyPSIX]

It wouldn't cause any flash/jump at all, just need to set css object-fit to contain (instead of cover), so all images with an incorrect/unknown aspect ratio will be scaled down instead of cropped

I think that would make images with aspect ratios > 1 get a lot of unwarranted empty space though

[12Me21]

It wouldn't cause any flash/jump at all, just need to set css object-fit to contain (instead of cover), so all images with an incorrect/unknown aspect ratio will be scaled down instead of cropped

I think that would make images with aspect ratios > 1 get a lot of unwarranted empty space though

On that note, I think a lot of people would really appreciate a setting to disable image cropping in general (in favor of letterboxing)

[WriterStat]

Tbh, in practical terms, no one would see or mind the shift because using a laptop, phone etc... you only see the first 2 or 3 posts, and not the ones below until you scroll, and the shift is fast enough to not be seen on the first 2 or 3.

If you were going to use bars, use the theme color, white for the light theme, black or dark for the dark theme? This seems like we are going backwards, making a fix for a problem that isn't really a problem.

[WriterStat]

Maybe give the user an option to choose either on their page, based on preference. Removing to go backwards doesn't seem the best. With that said. if it speeds up the servers then ok, but most software resizes images to fit in today's world or gives the option for letter box based on theme colors.

[eramdam]

Chiming in to add a little detail: @GlitchyPSIX and I noticed that, clearly, a lot (if not all?) of images posted by 3rd party clients/bots are affected since the aspectRatio field is not required on image records which would be fine... except it seems the PDS implementation doesn't compute those dimensions on image blobs?
Which surprises me because when I looked around in atproto/packages/pds I saw a bunch of code that looks like it should populate dimension metadata on image records?
https://github.com/bluesky-social/atproto/blob/main/packages/pds/src/actor-store/blob/transactor.ts#L45-L62
https://github.com/bluesky-social/atproto/blob/main/packages/pds/src/image/index.ts#L6-L39

It might just me misunderstanding or looking at the wrong thing though so maybe this is a red herring.

[eramdam]

since the aspectRatio field is not required on image records which

To expand on this, the fact that this field isn't required but also didn't exist until a few months ago makes me think the PDS would (should?) backfill that data if missing?
I'm just concerned because as far as 3rd party client/bot devs are concerned, this client-side change was a "breaking" change because an 'optional' field is now effectively required to provide a good experience. I almost would have preferred if the 'main API'¹ threw a "Bad request" error at me because it would have been a way to know about that change 😅

Footnotes

1. I know this isn't a thing really on AT Proto, but you get me ↩

[hopqer]

This issue is across all platforms for the official app as of 1.95 (I personally confirmed it on iOS, android, and both mobile web and desktop web). Third party bluesky apps like TOKIMEKI and deck blue are still displaying images normally.

[gaearon]

Yeah, I hear that it's frustrating that this isn't being enforced (fwiw enforcing it would have been a breaking change — which is impossible in Lexicon).

I also want to point out that as long as the client receives images without dimensions, with the old implementation, the layout shifts were inevitable which affect every post below the image in question in the feed. Layout shifts on scroll are inexcusable for a feed app. I totally hear the frustration but we needed to plug this perf/UX problem, and it is necessary for the clients to be aware that this field needs to be sent. So some kind of intervention was necessary.

I'll need to double check with the backend team whether there is any way those can be backfilled automatically. I would certainly prefer that myself too. My understanding of the constraints is that this was not possible, but I'll ask again.

[gaearon]

As for improving the current state, I'm working on a branch that displays them in a "contains" layout instead. I hope that addresses the worst of this.

We do need the third party clients to be aware that this field is important though.

[WriterStat]

You prolly already do all this, given this thought already... But in case helpful. We do the ratio on upload(server side) when not provided as you mentioned. And store in state. For images that don't have ratio for any reason, we do that ratio only on image request server side then send and store for next time. Or store and send. That way you don't have to do the entire back log at once or really ever. I'm guessing you do something similar/or already thought of this. I guess you could start a cron job to do it, but better to do on need/request just once.

[gaearon]

Potential fix / improvement PR #6828

[gaearon]

Re: server stuff, I'll have to sync with the backend team on this, but probably after the US holidays.

[WriterStat]

The only downside of using css object-fit: contain; is there will be a lot/a ton of unused blank space in the feed between text, horizontal images, and even posts in some cases in the posts. As a poster above mentioned. And in the light theme, an unusual large gray background of totally empty gray space between text and images. It's ok as a temp fix, till replaced.

But is unsightly in practical use, with lots of posts/in a feed, it is better than the current cropped out square that's there now, but not as good as the original functionality that was already in there prior to this change. I just tested it by changing all the page CSS, there is a lot of empty space between stuff on horizontal images. Note: I never saw the shifts in layout at any point, but truthfully users are used to dynamic content in a layout loading pretty much in web pages. IMHO contain isn't a long term fix.

[WriterStat]

I appreciate you and all of the bluesky team working hard to help and on this platform. And you working diligently on this. I know you are all pulled thin. So we appreciate it.

When you string a few posts into a feed using the CSS contained posts together in the light theme you will find that the feed may double/triple/quadruple etc... in length to read, for the same amount of posts and not look that good doing it. In the end, I'm just a user of the software so you'll need to do what's best for you. But this still feels like a change that should be rolled back until a good/true fix can be implemented/communicated to third party clients. Best, -C

[WriterStat]

Ok, bear with me, I have a thought:

If:

1. Roll the change back to the previous code. (that works)
2. Make the fields needed mandatory to third party posting services/api. (Or just make it a known rule.)
3. Let third party posting services know the fields are mandatory by rule.
4. The code will then only affect the display of old/older posts and not affect new ones that have the mandatory fields needed.
5. The old posts will work correctly as they were, until no longer needing to be displayed having aged out. Problem goes away by itself, as new posts are displayed and old posts are not displayed(but still can be if needed).
6. At some point the code being used will actually stop being used over a pretty short time period as the posts using it age away.

[WriterStat]

Here's my last post, I'm passionate about this platform, hope that's ok. =) You now have over 20 million users on the system you can't make system breaking changes for them without getting your third party client apps and posts submission and posts scheduling apps involved(communication and mandatory fields/data etc...).

Now you are forced into fixing it. Making a plan on the go/fly....

I like you. And the platform.

You are no longer small. Was this change that broke the system an emergency? Before the change? Were third party client apps and posts submission and posts scheduling apps let known of the changes beforehand now being mandatory so they could update their apps before the system breaking change?

You can't fix a communication issue with a client side tech fix. And one that really needs a tech fix to be fixed on the upload instead. You can only bandage it. Temp fix it or roll it back and then communicate.

Here's what my page looks like after the last system breaking change(basically unusable):

Here's what it will look like after the next layout breaking change:
It will add at least a third of blank empty space to every post with a horizontal image. Pages and feeds will be made at least a third again as visually long, possibly visually longer than they are now, and be chunky visually/chunked up by empty space. And feeds/pages will be a ton longer visually. But can be read.

Is there an emergency here needing this first system breaking change? Or can the code age out by providing the right field data in the post submission API? And communication? Or by a change in post upload processing?

I have made adjustments/workarounds to get my new posts to be viewable on my page.

So with the right communication to people the original problem breaking change if rolled back now ages out fast(possibly a week) and is only for old posts. But without the right communication this problem gets worse/bandaged.

Just some thoughts... Right now, all my older posts on my page are unusable visually basically. So usable but ugly is better than not usable. But still...

And with that, hope you all do well. Apologize for the passion. Best, -C

[ulrischa]

Hi Bluesky team. We (the bavarian environmental agency) joined bluesky and promote it. I'm a fan of your platform. But the change you introduced completely destroyed our posts: https://bsky.app/profile/lfu.bayern.de
We use a script to publish the postings to bluesky via your api. All went good but some time ago all images are displayed croped in square. And all old images are affected too. So our complete profile is scrambled. Please show a way how I can fix this

[gaearon]

I'll double-check with the backend crew if it's possible to somehow fill these in from the server instead of relying on the aspectRatio in the uploaded post records. I agree the current situation is unfortunate.

I am fairly certain that we're not going to roll back this change — it has to be done at some point because we can't tolerate layout shifts (jumping content) in feeds, and having even a single post like this in a feed causes everything below it to jump. Having multiple means multiple jumps. This causes performance issues for everyone else — having a single post in a feed like this is enough to hurt all content that appears below it.

We just can't keep doing it like this.

We do plan to deploy #6828 soon which, while not ideal, should improve how images with missing dimensions are displayed.

I do agree we want to emphasize this field in the documentation. I would really appreciate pointers on which pages you'd find it most helpful to see mentioned. AFAIK we're not able to make this an actually required field because lexicons have to be backwards-compatible. I.e. we can't actually break old posts. So that is not an option. But we can strongly recommend to include it.

Re: communication, for a change like this, it is pretty much impossible to communicate beforehand and hope that everybody will hear about it. Most major clients I'm aware of (maybe all by now?) have already been sending the image dimensions correctly, but there will always be a long tail of one-off scripts, libraries, etc. For better or worse, displaying them suboptimally in the app is the most effective way to actually ensure that the change gets noticed. I 100% agree it's frustrating for older posts, but delaying the change is even worse because even more posts will be made before the change takes effect the next time. That's the motivation for ripping the bandaid.

But again, I also fully agree that ideally this should not have been an issue in the first place. I hope that this will be solvable on the backend, so I'll check on that. For now, #6828 and documentation updates (please let me know if you see a page that needs to be changed) would hopefully improve this.

Thanks!

[ulrischa]

@gaearon thank you for your detailed explanation. #6828 would be very necessary to be rolled out as soon as possible. I don't want to sound rude, but for a platform that promotes maximum openness such a breaking change not communicated is not very trustworthy. You can not break functionality based on an optional api field.

[WriterStat]

I like you all. Do the 3rd party apps have to nominally register to use your API to submit posts for people to the system?

If not, then that should be fixed right away. Because they should. And it would solve your communication to them easily/knowing who to communicate to/problems to, them being registered. If they don't have to isn't that in and of itself an issue?

The rest... I can see you are running down that path. It hurts me to see it. It especially hurt my page. =) There are so many alternatives.

Can't API changes be date dependent? Also really curious.
I get it.

[mary-ext]

there is no versioning and migrations cannot happen because AT Protocol is a generic data store, the only mitigation for this is if Bluesky's AppView can ask the CDN to fill in the missing metadata instead of serving as is.

though I think it'd still be preferred if bots/clients were to properly fill this metadata by themselves, instead of relying on this (soon to be implemented) missing fill behavior.

I can communicate this change to the Bluesky API Touchers Discord server if an announcement is needed. this has been announced.

[gaearon]

#6828 should be live on the web, we'll get it to native apps soon as well.

[gaearon]

I've checked with the backend crew — we should be able to eventually solve this in the future (even for old posts) by changing how AppView is involved with the CDN but not something we can do quickly right now.

[WriterStat]

Cool!

[ulrischa]

@gaearon thanks for the fast fix

[mistydemeo]

I appreciate that displaying images poorly is an approach that's being taken to communicate the change, but as Damien points out, there are accounts that have been creating these kinds of images since before the field existed. For accounts such as one I run or Damien's, the idea that this older content is simply permanently broken and it will never be possible to fix it is... frustrating? Suboptimal? Especially since people do regularly go back to older posts on these accounts, and so their media is less ephemeral than the average post. It doesn't make me feel good about any future changes that might happen. If backfilling this data for older posts is possible, it would be really appreciated.

[gaearon]

The proper fix for this is to rewire how our server communicates with the CDN, at which point this will become a non-issue and would work for old images too. Right now we're in an unfortunate transient state where we can't fix it properly yet.

[gaearon]

Well, to be clear, I think this can be made to work for old images too, can't promise that yet.

Maybe we can explore some compromises like accepting layout shifts in profiles (e.g. when visiting old posts) but using the current behavior in feeds. Then at least perf loss would be contained.

[gaearon]

Improving the docs bluesky-social/bsky-docs#290