feat(media): Blurhash #1381

sorin-davidoi · 2019-08-08T16:08:38Z

Todo:

Handle medias without blurhash
Look into performance (maybe cache blurhash -> base64 computation)
Implement for videos
Fix integration tests
Option to disable usage

Closes #1179.

sorin-davidoi · 2019-08-08T16:11:04Z

src/routes/_utils/blurhash.js

+  const pixels = decodeBlurHash(blurhash, 320, 320)
+
+  if (pixels) {
+    canvas = canvas || document.createElement('canvas')


Mastodon renders the canvases directly but I though that it's too much overhead to create and destroy that many canvases so I went with this approach.

Profile sample with 6x CPU throttling.

I wonder if toBlob() with a Blob URL may actually be more performant here than a data URL. I'll play around with it. :)

Hm, so the problem with toBlob() is it's asynchronous, meaning that there might be a flash of unhidden sensistive media unless we're careful about doing the blurhashing in advance. I might keep noodling on this after merging the PR.

sorin-davidoi · 2019-08-09T05:46:03Z

package.json

@@ -48,6 +48,7 @@
    "@webcomponents/custom-elements": "^1.2.4",
    "babel-loader": "^8.0.6",
    "babel-plugin-transform-react-remove-prop-types": "^0.4.24",
+    "blurhash": "^1.1.3",


We could try https://github.com/fpapado/blurhash-rust-wasm if performance turns out to be an issue.

sorin-davidoi · 2019-08-09T15:45:14Z

src/routes/_utils/blurhash.js

+import { decode as decodeBlurHash } from 'blurhash'
+import { mark, stop } from './marks'
+
+const RESOLUTION = 32


We could maybe lower this based on some heuristics (e.g. navigator.hardwareConcurrency)? Setting it to 1 will basically extract the average color, so maybe we can do that initially and repeat with 32 inside a requestIdleCallback?

I'm not sure what the limiting factor is for blurhash, but yeah for now a hard-coded value is probably fine.

nolanlawson · 2019-08-10T17:51:56Z

Thanks very much for this PR!! I made some minor refactoring changes. At this point, my only major concerns are:

This doesn't work with "show large inline images." Although honestly maybe it's time to just remove that option; I'm not sure how many people are using it, given that @sgenoud's new image system is so much better than the old one.
In the settings, "Use blurhash for previews" is now the only option that is checked by default. Up to now I have tried to prefer "false by default" to reduce cognitive load on users (i.e. so they can remember what the defaults are). I admit though that it's hard to think of a negation for this feature that wouldn't be awkward. ("Hide blurhash"?)
Yeah the performance of blurhashing is a bit concerning, especially when I turn on 6x slowdown in Chrome (up to 100ms, dominates the time spent rendering a status overall). I think long term though, we could move toward a webworker-based solution or toBlob() solution, but that would probably require doing some initial asynchronous work before rendering (e.g. in VirtualListLazyItem.html). I've been meaning to do that for some other stuff anyway. But that can wait for a future PR.

For now I'm mostly concerned about # 1; maybe it's time for me to retire the "show large inline images system." Or we could implement blurhash for it.

nolanlawson · 2019-08-10T17:56:07Z

Created a poll to see if anyone is using this setting: https://mastodon.technology/@pinafore/102593964013067092

sorin-davidoi · 2019-08-10T21:47:21Z

Pushed a tentative fix for the overflow issue when "show large inline images" is enabled.

nolanlawson · 2019-08-11T18:13:18Z

Awesome! In that case I'd say remaining tasks are:

Negate the setting somehow... maybe "Show a plain gray color for sensitive media" ?
Fix the performance issues... maybe indeed blurhash-rust-wasm is the best fix here

sorin-davidoi · 2019-08-11T20:15:20Z

Can't seem to get blurhash-rust-wasm to play along (getting Error: Cannot find module 'blurhash-wasm').

sorin-davidoi · 2019-08-14T23:00:36Z

Pushed a commit where the computations are moved to a Web Worker:

if OffscreenCanvas is supported the main thread sends the encoded string and gets back an URL
otherwise it sends the encoded string and gets back an ImageData, which it then renders using the canvas

Some things still need to be addresses though:

the Web Worker is created lazily (when the first blurhash needs to be computed) - this introduces some delay, we should call init as soon as we know that blurhashes are not disabled (when the store is created?)
some caching for URLs should probably be added, especially since we currently don't ever call URL.revokeObjectURL

sorin-davidoi · 2019-08-14T23:02:14Z

src/routes/_components/status/Media.html

@@ -141,6 +166,7 @@
    },
    data: () => ({
      oneTransparentPixel: ONE_TRANSPARENT_PIXEL,
+      decodedBlurhash: ONE_TRANSPARENT_PIXEL,


ONE_TRANSPARENT_PIXEL is used while the async computation is pending.

Good workaround!

sorin-davidoi · 2019-08-14T23:03:37Z

src/routes/_utils/blurhash.js

@@ -0,0 +1,49 @@
+import BlurhashWorker from 'worker-loader!../_workers/blurhash' // eslint-disable-line


I couldn't get worker-loader working when added to client.config.js for some reason. Standard complains about the inline use of the loader, disabled it for now.

That's fine; I don't really care whether the worker-loader is referenced in the Webpack config or in the import.

sorin-davidoi · 2019-08-15T07:54:22Z

Added caching, but we are still leaking memory due to never calling revokeObjectURL - not sure when would be the best time to call it.

Throws exception which prevents the image from loading.

sorin-davidoi · 2019-08-15T16:14:47Z

There are still some overflow issues caused by padding-bottom: 56.25%:

nolanlawson · 2019-08-17T17:54:34Z

Fantastic work, this looks great to me. I only have some minor issues remaining:

The eye icon and "sensitive media" text are a bit hard to read; I think the font should be lighter when blurhash is enabled
It would be nice to do the worker operations at the same time that the content is fetched asynchronously from IDB so that there isn't a "pop" when the color changes from gray to the blurhash (i.e. delay rendering the toot until the blurhash is ready... it's a tradeoff I'd prefer to make in favor of less visually jarring rendering).

Both of these things can be fixed later, though; overall I think this is a fantastic PR and we should just get it in master and fix the minor things later. Thanks for the excellent work!!

sorin-davidoi added 2 commits August 8, 2019 18:06

chore(npm): Install blurhash

75b7ab6

feat(media): Show blurhash

7dd3ac2

sorin-davidoi commented Aug 8, 2019

View reviewed changes

sorin-davidoi added 5 commits August 8, 2019 18:47

fix(media/blurhash): Better sensitive video handling

219ff37

feat(media): Preference for using blurhash

4dc5a46

chore(utils/blurhash): Add performance marks

ddef2dd

fix(utils/blurhash): Performance marks

2bebfdd

fix(utils/blurhash): Use correct dimension

a1f5096

sorin-davidoi commented Aug 9, 2019

View reviewed changes

refactor(utils/blurhash): Use constant for number of pixels

dcb0c69

sorin-davidoi commented Aug 9, 2019

View reviewed changes

sorin-davidoi added 7 commits August 9, 2019 17:49

refactor(media): Simplify logic for displaying blurhash

fdef73b

chore(tests/spec): Attempt to adjust sensitivity tests for blurhash

0562e66

chore(tests/spec): Update sensitivity tests for blurhash

f8f479d

chore(tests/spec): Check for sensitive

8f25d11

fix(media/blurhash): Handle videos

5aa04d6

fix: Video handling

f080728

fix: Videos

ba9c396

sorin-davidoi changed the title ~~[WIP] feat(media): Blurhash~~ feat(media): Blurhash Aug 9, 2019

minor refactoring, fix Svelte warning

557176f

fix: Large inline images and videos

dc1d11e

feat(settings): Rename blurhash setting

f0306e5

sorin-davidoi added 2 commits August 15, 2019 00:14

refactor: Use toBlob, block media rendering until blurhash ready

7af9278

refactor: Move computations to Web Worker

39c24ea

sorin-davidoi commented Aug 14, 2019

View reviewed changes

sorin-davidoi added 4 commits August 15, 2019 09:12

fix(workers/blurhash): More error handling

20a6ee8

feat(workers/blurhash): Use quick-lru for caching

66c829a

fix: Don't create Context2D needlessly

421af34

fix(workers/blurhash): Increase cache size to 100

b711fb6

sorin-davidoi added 2 commits August 15, 2019 12:23

fix(workers/blurhash): Don't resolve promise twice

fa00e04

fix(utils/decode-image): Ignore data URLs

6ec0d28

Throws exception which prevents the image from loading.

nolanlawson merged commit 77bb784 into nolanlawson:master Aug 17, 2019

This was referenced Aug 17, 2019

fix: tweak colors for blurhash sensitive media button/text #1389

Merged

fix: move blurhash worker operations to before status rendering #1391

Merged

Call revokeObjectURL on stale blurhash URLs #1402

Closed

sorin-davidoi mentioned this pull request Aug 18, 2019

Handling of grouped images with blurhash is broken #1404

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(media): Blurhash #1381

feat(media): Blurhash #1381

sorin-davidoi commented Aug 8, 2019 •

edited

sorin-davidoi Aug 8, 2019 •

edited

sorin-davidoi Aug 9, 2019

nolanlawson Aug 10, 2019

nolanlawson Aug 10, 2019

sorin-davidoi Aug 9, 2019

sorin-davidoi Aug 9, 2019

nolanlawson Aug 10, 2019

nolanlawson commented Aug 10, 2019

nolanlawson commented Aug 10, 2019

sorin-davidoi commented Aug 10, 2019

nolanlawson commented Aug 11, 2019

sorin-davidoi commented Aug 11, 2019

sorin-davidoi commented Aug 14, 2019

sorin-davidoi Aug 14, 2019

nolanlawson Aug 17, 2019

sorin-davidoi Aug 14, 2019

nolanlawson Aug 17, 2019

sorin-davidoi commented Aug 15, 2019

sorin-davidoi commented Aug 15, 2019

nolanlawson commented Aug 17, 2019

		@@ -0,0 +1,49 @@
		import BlurhashWorker from 'worker-loader!../_workers/blurhash' // eslint-disable-line

feat(media): Blurhash #1381

feat(media): Blurhash #1381

Conversation

sorin-davidoi commented Aug 8, 2019 • edited

sorin-davidoi Aug 8, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nolanlawson commented Aug 10, 2019

nolanlawson commented Aug 10, 2019

sorin-davidoi commented Aug 10, 2019

nolanlawson commented Aug 11, 2019

sorin-davidoi commented Aug 11, 2019

sorin-davidoi commented Aug 14, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sorin-davidoi commented Aug 15, 2019

sorin-davidoi commented Aug 15, 2019

nolanlawson commented Aug 17, 2019

sorin-davidoi commented Aug 8, 2019 •

edited

sorin-davidoi Aug 8, 2019 •

edited