Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhance: Proxy custom emojis to reduce image size and accelerate the frontend #9431

Merged
merged 15 commits into from Dec 30, 2022

Conversation

tamaina
Copy link
Member

@tamaina tamaina commented Dec 29, 2022

Resolve #8319
Fix #9413

カスタム絵文字をプロキシし、画像サイズを縮小することでクライアントの軽量化を狙います。

What

  1. /proxy(MediaProxyServerService)にemojiモードを追加
    高さ128px(引き伸ばしなし)のwebpを返すように
  2. /emoji/:path.webpは全て/proxy/?emoji=1に301リダイレクトするように
  3. CustomEmojiService.populateEmojiのurlはプロキシされていないURLが返ってくるように Fix 絵文字のプロキシURLの改善 #9413
    ( refactor: 絵文字URLを引き回すのをやめる #9423 によってMisskey Webではこれを使わなくなったため + /proxy/?emoji=1 を使えばなんとかなるため )
  4. convertToWebp / convertSharpToWebp の仕様変更
  5. backend/prelude/url, frontend/scripts/url, sw/scripts/urlにコメント追加
  6. frontend/scripts/get-static-image-urlをfrontend/scripts/media-proxyに統合
  7. MkAutocompleteでMkEmojiを使うように、型エラー修正
  8. animated webpを扱うためにsharp.jsをアップデート
  9. production以外でtmpをcleanupしないように

@tamaina tamaina requested a review from syuilo December 29, 2022 17:35
@github-actions github-actions bot added packages/backend Server side specific issue/PR packages/frontend Client side specific issue/PR labels Dec 29, 2022
@tamaina
Copy link
Member Author

tamaina commented Dec 29, 2022

アニメーション画像がおかしなことになる…

image

@codecov
Copy link

codecov bot commented Dec 29, 2022

Codecov Report

Merging #9431 (146df69) into develop (dac4f68) will increase coverage by 0.00%.
The diff coverage is 28.84%.

@@           Coverage Diff            @@
##           develop    #9431   +/-   ##
========================================
  Coverage    22.40%   22.41%           
========================================
  Files          704      704           
  Lines        66070    66098   +28     
  Branches      2134     2134           
========================================
+ Hits         14801    14813   +12     
- Misses       51269    51285   +16     
Impacted Files Coverage Δ
packages/backend/src/core/CustomEmojiService.ts 39.32% <0.00%> (-1.66%) ⬇️
packages/backend/src/core/DownloadService.ts 31.00% <0.00%> (ø)
packages/backend/src/misc/create-temp.ts 20.83% <0.00%> (ø)
...ages/backend/src/server/MediaProxyServerService.ts 0.00% <0.00%> (ø)
...ages/backend/src/server/web/ClientServerService.ts 0.00% <0.00%> (ø)
...ackages/backend/src/core/ImageProcessingService.ts 47.36% <71.42%> (+5.85%) ⬆️
packages/backend/src/misc/prelude/url.ts 44.44% <100.00%> (+21.36%) ⬆️
packages/backend/src/misc/secure-rndstr.ts 100.00% <0.00%> (+9.52%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@tamaina
Copy link
Member Author

tamaina commented Dec 29, 2022

sharpのバージョンが古いせいだった

@tamaina
Copy link
Member Author

tamaina commented Dec 29, 2022

アニメーションGIFからアニメーションWebPに変換するとサイズが大きくなってしまう

181.65kB
https://s3.arkjp.net/misskey/d10569e6-047b-45b4-a038-bef6df5e6254.gif

223.29kB
https://p1.a9z.dev/proxy/emoji.webp?url=https%3A%2F%2Fs3.arkjp.net%2Fmisskey%2Fd10569e6-047b-45b4-a038-bef6df5e6254.gif&emoji=1

@tamaina
Copy link
Member Author

tamaina commented Dec 29, 2022

@tamaina
Copy link
Member Author

tamaina commented Dec 29, 2022

mixed: trueでめちゃんこ圧縮されるらしいがslowと書いてあるのが気になる

https://sharp.pixelplumbing.com/api-output#webp

@tamaina
Copy link
Member Author

tamaina commented Dec 29, 2022

@tamaina
Copy link
Member Author

tamaina commented Dec 30, 2022

MkAutocompleteでMkEmojiを使うようにした

@syuilo syuilo merged commit 8b46ede into misskey-dev:develop Dec 30, 2022
@syuilo
Copy link
Member

syuilo commented Dec 30, 2022

👍

syuilo pushed a commit that referenced this pull request Dec 30, 2022
* fix(server): /emoji to accept `@.` host expression

* fix(client): use MkEmoji for custom emoji in MkEmojiPicker

* change convertToWebp

* nanka iroiro

* remove

* fix

* nearLosslessは労多くして益少なしなのでやめる

* do not cleanup tmp for development

* update sharp.js to 0.31.3

* mixed: true

* fix MkAutocomplete of 912791b

* clean up

* #9431 (comment)

* edit mkautocomplete.vue use sanitize-html

* fix package.json
@tamaina
Copy link
Member Author

tamaina commented Dec 30, 2022

めすいきー系かv10系かの絵文字が表示できていないような気がするけどおま環かもしれない

pikokr added a commit to pikokr/misskey that referenced this pull request Jan 2, 2023
commit f4b012b
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Mon Jan 2 10:21:31 2023 +0900

    13.0.0-beta.15

commit 7a95339
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Mon Jan 2 10:18:47 2023 +0900

    enhance(client): user activity page

commit 1df23a8
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Mon Jan 2 09:52:10 2023 +0900

    fix style

commit 9f3b939
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Mon Jan 2 09:43:41 2023 +0900

    clean up

commit d244622
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Mon Jan 2 09:41:43 2023 +0900

    :art:

commit faa25fa
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Mon Jan 2 09:39:18 2023 +0900

    fix container query style

commit 2e3e49e
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Mon Jan 2 09:36:19 2023 +0900

    :art:

commit 7501609
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Mon Jan 2 09:33:37 2023 +0900

    :art:

commit 53afc68
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Mon Jan 2 09:21:44 2023 +0900

    :art:

commit 8bc0aa3
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Mon Jan 2 09:20:49 2023 +0900

    :art:

commit a4a614e
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sun Jan 1 18:00:06 2023 +0900

    13.0.0-beta.14

commit 969e9df
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sun Jan 1 17:45:49 2023 +0900

    feat: add per user pv chart

commit 4c4af2a
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sun Jan 1 17:38:02 2023 +0900

    clean up

commit c3a3669
Author: tamaina <tamaina@hotmail.co.jp>
Date:   Sun Jan 1 17:11:33 2023 +0900

    use Intl.DateTimeFormat and Intl.NumberFormat instead of
toLocaleString (misskey-dev#9444)

commit 0222165
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sun Jan 1 17:01:11 2023 +0900

    13.0.0-beta.13

commit 7f37415
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sun Jan 1 16:57:13 2023 +0900

    :art:

commit e414737
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sun Jan 1 16:53:10 2023 +0900

    feat: make possible to export favorited notes

    misskey-dev#9331

commit 0d34d28
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sun Jan 1 14:12:25 2023 +0900

    :art:

commit 527fe90
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sun Jan 1 12:28:30 2023 +0900

    :art:

commit ee6e022
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sun Jan 1 11:58:16 2023 +0900

    enhance(server): productionじゃない限りレートリミットを無効に

commit 86e8864
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sun Jan 1 11:57:37 2023 +0900

    :art:

commit 5004304
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sun Jan 1 09:45:27 2023 +0900

    :art:

commit 6f51e73
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sun Jan 1 09:35:14 2023 +0900

    :v:

commit 37ef9b7
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sat Dec 31 21:10:25 2022 +0900

    :art:

commit f5b8ec5
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sat Dec 31 20:41:26 2022 +0900

    13.0.0-beta.12

commit 24bfc77
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sat Dec 31 20:41:14 2022 +0900

    New translations ja-JP.yml (Korean) (misskey-dev#9436)

commit 1cc931c
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sat Dec 31 20:40:47 2022 +0900

    :art:

commit b64daa5
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sat Dec 31 20:36:49 2022 +0900

    :art:

commit 3e508d7
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sat Dec 31 20:14:24 2022 +0900

    :art:

commit 40dddd8
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sat Dec 31 19:46:35 2022 +0900

    13.0.0-beta.11

commit c05ad89
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sat Dec 31 19:46:16 2022 +0900

    :art:

commit aba06b4
Merge: 5703792 0f1091e
Author: tamaina <tamaina@hotmail.co.jp>
Date:   Sat Dec 31 06:18:32 2022 +0000

    Merge branch 'develop' of https://github.com/misskey-dev/misskey
into develop

commit 5703792
Author: tamaina <tamaina@hotmail.co.jp>
Date:   Sat Dec 31 06:18:23 2022 +0000

    Fix misskey-dev#9438

commit 0f1091e
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sat Dec 31 13:18:27 2022 +0900

    :art:

commit cb237a0
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sat Dec 31 13:18:27 2022 +0900

    :art:

commit d52e8cf
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sat Dec 31 09:25:15 2022 +0900

    13.0.0-beta.10

commit ced29ce
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sat Dec 31 09:20:36 2022 +0900

    New Crowdin updates (misskey-dev#9421)

    * New translations ja-JP.yml (Slovak)

    * New translations ja-JP.yml (Polish)

    * New translations ja-JP.yml (Ukrainian)

    * New translations ja-JP.yml (Ukrainian)

    * New translations ja-JP.yml (Ukrainian)

    * New translations ja-JP.yml (Korean)

    * New translations ja-JP.yml (Chinese Traditional)

    * New translations ja-JP.yml (German)

    * New translations ja-JP.yml (English)

    * New translations ja-JP.yml (Chinese Simplified)

    * New translations ja-JP.yml (Thai)

    * New translations ja-JP.yml (Slovak)

    * New translations ja-JP.yml (Thai)

    * New translations ja-JP.yml (German)

    * New translations ja-JP.yml (English)

    * New translations ja-JP.yml (Thai)

    * New translations ja-JP.yml (Thai)

    * New translations ja-JP.yml (Japanese, Kansai)

    * New translations ja-JP.yml (Slovak)

commit 6140865
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sat Dec 31 09:14:53 2022 +0900

    tweak max-age

commit 51d8d2a
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sat Dec 31 09:12:42 2022 +0900

    cache emoji response

commit 6f648fc
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sat Dec 31 08:51:35 2022 +0900

    clean up

commit 7157fd6
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sat Dec 31 08:46:27 2022 +0900

    fix(server): driveCapacityOverrideMbが公開情報になっているのを修正

commit b2d6561
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sat Dec 31 08:43:13 2022 +0900

    enhance(server): clean emoji response

commit fcfcb8d
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sat Dec 31 08:18:58 2022 +0900

    13.0.0-beta.9

commit e1846e2
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Sat Dec 31 08:17:00 2022 +0900

    :art:

commit 1a408db
Author: tamaina <tamaina@hotmail.co.jp>
Date:   Fri Dec 30 13:37:58 2022 +0000

    fix: use || for emoji url fallback instead of ??

commit 8090850
Author: tamaina <tamaina@hotmail.co.jp>
Date:   Fri Dec 30 10:48:05 2022 +0000

    import sanitizeHtml from 'sanitize-html';

commit 2d89741
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Fri Dec 30 19:01:01 2022 +0900

    ReactionsViewerが消えることがある問題を修正

commit c631e14
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Fri Dec 30 18:33:26 2022 +0900

    :art:

commit 3968597
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Fri Dec 30 18:26:36 2022 +0900

    improve heatmap

commit 78a8e91
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Fri Dec 30 17:34:42 2022 +0900

    :art:

commit c72ee0a
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Fri Dec 30 15:41:41 2022 +0900

    remove console.log

commit d147181
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Fri Dec 30 15:40:29 2022 +0900

    :art:

commit 43fafc8
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Fri Dec 30 14:24:00 2022 +0900

    enhance(client): tweak url preview

commit 4767804
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Fri Dec 30 14:10:00 2022 +0900

    :art:

commit 2244cf2
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Fri Dec 30 13:56:22 2022 +0900

    refactor

commit 93d3cc3
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Fri Dec 30 13:52:40 2022 +0900

    :art:

commit 632c7d2
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Fri Dec 30 13:37:14 2022 +0900

    refactor

commit 254fede
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Fri Dec 30 13:29:30 2022 +0900

    :art:

commit e023a9b
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Fri Dec 30 13:27:37 2022 +0900

    :art:

commit 87a0ed8
Author: tamaina <tamaina@hotmail.co.jp>
Date:   Fri Dec 30 13:02:18 2022 +0900

    fix(client): add sanitize-html to MkAutocomplete (misskey-dev#9433)

    * fix(server): /emoji to accept `@.` host expression

    * fix(client): use MkEmoji for custom emoji in MkEmojiPicker

    * change convertToWebp

    * nanka iroiro

    * remove

    * fix

    * nearLosslessは労多くして益少なしなのでやめる

    * do not cleanup tmp for development

    * update sharp.js to 0.31.3

    * mixed: true

    * fix MkAutocomplete of 912791b

    * clean up

    *
misskey-dev#9431 (comment)

    * edit mkautocomplete.vue use sanitize-html

    * fix package.json

commit f213f30
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Fri Dec 30 12:01:29 2022 +0900

    13.0.0-beta.8

commit 8b46ede
Author: tamaina <tamaina@hotmail.co.jp>
Date:   Fri Dec 30 12:00:50 2022 +0900

    enhance: Proxy custom emojis to reduce image size and accelerate the
frontend (misskey-dev#9431)

    * fix(server): /emoji to accept `@.` host expression

    * fix(client): use MkEmoji for custom emoji in MkEmojiPicker

    * change convertToWebp

    * nanka iroiro

    * remove

    * fix

    * nearLosslessは労多くして益少なしなのでやめる

    * do not cleanup tmp for development

    * update sharp.js to 0.31.3

    * mixed: true

    * fix MkAutocomplete of 912791b

    * clean up

    *
misskey-dev#9431 (comment)

commit f227091
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Fri Dec 30 11:32:45 2022 +0900

    fix(client): fix position calculation of nested context menu

commit dac4f68
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Fri Dec 30 11:14:11 2022 +0900

    fix of 912791b

commit ec64a99
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Fri Dec 30 11:10:38 2022 +0900

    fix of 912791b

commit f528d17
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Fri Dec 30 11:06:38 2022 +0900

    fix of 912791b

commit 198123f
Merge: dcfc526 e35e788
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Fri Dec 30 08:41:26 2022 +0900

    Merge branch 'develop' of https://github.com/misskey-dev/misskey
into develop

commit dcfc526
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Fri Dec 30 08:41:22 2022 +0900

    fix 912791b

commit e35e788
Author: tamaina <tamaina@hotmail.co.jp>
Date:   Thu Dec 29 15:15:30 2022 +0000

    fix(client): use MkEmoji for custom emoji in MkEmojiPicker

commit 329ef76
Author: tamaina <tamaina@hotmail.co.jp>
Date:   Thu Dec 29 14:54:31 2022 +0000

    fix(server): /emoji to accept `@.` host expression

commit 09078de
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Thu Dec 29 17:33:44 2022 +0900

    13.0.0-beta.7

commit cbbf1a6
Author: rinsuki <428rinsuki+git@gmail.com>
Date:   Thu Dec 29 17:26:33 2022 +0900

    chore: add message about server specific issue (misskey-dev#9195)

    * chore: add message about misskey.io specific issue

    * Update 01_bug-report.md

    * Update 01_bug-report.md

commit 0d571f5
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Thu Dec 29 17:26:10 2022 +0900

    :art:

commit 352e16b
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Thu Dec 29 17:24:46 2022 +0900

    :art:

commit e611250
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Thu Dec 29 17:20:24 2022 +0900

    通知設定が開けないのを修正

commit 666c9be
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Thu Dec 29 17:16:35 2022 +0900

    :art:

commit d031f64
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Thu Dec 29 10:18:13 2022 +0900

    13.0.0-beta.6

commit 912791b
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Thu Dec 29 10:14:44 2022 +0900

    refactor: 絵文字URLを引き回すのをやめる (misskey-dev#9423)

commit 510e6ec
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Thu Dec 29 09:58:11 2022 +0900

    :art:

commit d19dab8
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Thu Dec 29 09:36:41 2022 +0900

    fix style

commit 845c202
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Thu Dec 29 09:31:49 2022 +0900

    fix(client): オートコンプリートが出ない問題を修正

commit 5739427
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Thu Dec 29 07:55:20 2022 +0900

    Update CHANGELOG.md

commit 123e490
Author: Soni L <EnderMoneyMod@gmail.com>
Date:   Wed Dec 28 19:53:59 2022 -0300

    Support OpenSearch (closes misskey-dev#7058) (misskey-dev#9427)

    Co-authored-by: Chaos <chaoticryptidz@owo.monster>

commit c4e4ffc
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Wed Dec 28 18:28:48 2022 +0900

    13.0.0-beta.5

commit 25c8d77
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Wed Dec 28 18:28:01 2022 +0900

    update deps

commit 6e6d9f3
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Wed Dec 28 18:22:46 2022 +0900

    :art:

commit 229d185
Author: syuilo <Syuilotan@yahoo.co.jp>
Date:   Wed Dec 28 18:02:11 2022 +0900

    :art:
@syuilo
Copy link
Member

syuilo commented Jan 6, 2023

一部のアニメーション画像が動かなくなるっぽい

syuilo added a commit that referenced this pull request Jan 6, 2023
@tamaina
Copy link
Member Author

tamaina commented Jan 6, 2023

例えばどれ?

@syuilo
Copy link
Member

syuilo commented Jan 6, 2023

例えばどれ?

https://p1.a9z.dev/notes/99n31yxjgr

@tamaina
Copy link
Member Author

tamaina commented Jan 6, 2023

495d513

これ外すとクライアントが重たいので差し戻すのはやめて欲しかった

@syuilo
Copy link
Member

syuilo commented Jan 6, 2023

アニメーション画像が表示されないのは問題

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
packages/backend Server side specific issue/PR packages/frontend Client side specific issue/PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

絵文字のプロキシURLの改善 カスタム絵文字はサムネイルで十分そう
2 participants