Add brotli compression for text-based inscriptions #1713

terror · 2023-02-13T02:23:03Z

Tests:

content encoding tag is serialized corretly. (just create inscription with content encoding field and turn it into an envelope)
test that content encoding appears on /inscription page
add unit tests for acceptencodingbrotli (test that it's correct with multiple values, spaces, qvalues, etc)
allow content encodings other than brotli (may need to get header value, and check if inscription.content_encoding() is present, instead of using AcceptEncodingBrotli)
inscriptions are compressed when passing the --compress flag
inscriptions are not compressed if compression doesn't make content smaller
test that when getting /content, server returns 406 if no content encoding header is passed when inscription is compressed
test that when getting /preview, server returns 406 if no content encoding header is passed when inscription is compressed

casey · 2023-02-13T02:53:36Z

You can add a bool to Media::TABLE, true for text formats and false for non text formats, which determines what should be compressed. Only text should be compressed for now.

Psifour · 2023-02-13T04:28:06Z

Only text should be compressed for now.

I had tested the majority of our existing MIME types to verify which would benefit from brotli. The majority show gains with the 3d types being exceptionally well suited for compression.

src/inscription.rs

nammaki · 2023-03-23T18:09:27Z

I hope this will be merged someday 🙏

src/envelope.rs

src/inscription.rs

src/media.rs

src/inscription.rs

src/subcommand/server/accept_encoding.rs

src/templates/inscription.rs

src/test.rs

static/preview-text.js

tests/wallet/inscribe.rs

casey

Very minor nits, plus looks like tests and lint are failing.

src/subcommand/server.rs

src/subcommand/server/accept_encoding.rs

terror · 2023-11-15T03:56:55Z

TODO (to get this shipped):

Switch to dropbox rust brotli implementation (https://docs.rs/brotli/latest/brotli/)
Handle these tests:
- subcommand::server::tests::responses_are_brotlied
- subcommand::server::tests::text_preview
- subcommand::server::tests::text_preview_returns_error_when_content_is_not_utf8
- subcommand::server::tests::text_preview_text_is_escaped

casey

🗜️

* Fix lost sats bug (ordinals#2666) * Add Hindi version of handbook (ordinals#2648) * Remove Index::index_block_inscription_numbers (ordinals#2667) * Hide protocol inscriptions (ordinals#2674) * Don't color links in headers (ordinals#2678) * Add inscription charms (ordinals#2681) * Group rune server tests (ordinals#2685) * Add inscription compression (ordinals#1713) * Fix media table formatting (ordinals#2686) * Update schema version for charms (ordinals#2687) * Fix unbound outpoint server error (ordinals#2479) * Add binary media type (ordinals#2671) * Clean up install.sh (ordinals#2669) * Add /collections Page (ordinals#2561) * Preview font inscriptions (ordinals#2692) * Only load used language highlight module in code preview (ordinals#2696) * Only try to create the database if it wasn't found (ordinals#2703) * Move postage into batch file (ordinals#2705) * Add destination field to batch (ordinals#2701) * Use sequence numbers database keys (ordinals#2664) * Update redb to 1.4.0 (ordinals#2714) * Refactor inscriptions paginations (ordinals#2715) * Display table stats in `ord index info` (ordinals#2711) * Use redb's recovery callback API (ordinals#2584) * Allow setting CSP origin (ordinals#2708) * Remove default file path from `ord index export --tsv` (ordinals#2717) * Use icons in nav bar (ordinals#2722) * Add Debian packaging instructions (ordinals#2725) * Add Homebrew install instructions to readme (ordinals#2726) * Add sat recursive endpoints with index and pagination (ordinals#2680) * Only accept sat number in recursive endpoint (ordinals#2732) * Fix typo in docs/src/inscriptions/metadata.md (ordinals#2731) * Add docs for metadata recursive endpoint (ordinals#2734) * Remove `RUNE` from <h1> on /rune (ordinals#2728) * Add /r/children recursive endpoint (ordinals#2431) * Add docs and examples for sat recursive endpoint (ordinals#2735) * Ignore flaky test (ordinals#2742) * Update docs to include all fields, including content-encoding (ordinals#2740) * Add docs for child recursive endpoint (ordinals#2743) * Hide JSON and .btc (ordinals#2744) * Release 0.12.0 (ordinals#2746) --------- Co-authored-by: raph <raphjaph@protonmail.com> Co-authored-by: duttydeedz <142775511+duttydeedz@users.noreply.github.com> Co-authored-by: Casey Rodarmor <casey@rodarmor.com> Co-authored-by: liam <31192478+terror@users.noreply.github.com> Co-authored-by: Eloc <42568538+elocremarc@users.noreply.github.com> Co-authored-by: Julian Eager <eagr@tutanota.com> Co-authored-by: ordinally <11798624+veryordinally@users.noreply.github.com> Co-authored-by: Christopher Berner <me@cberner.com> Co-authored-by: Rijndael <115941166+rot13maxi@users.noreply.github.com> Co-authored-by: vuittont60 <81072379+vuittont60@users.noreply.github.com>

* Fix lost sats bug (ordinals#2666) * Add Hindi version of handbook (ordinals#2648) * Remove Index::index_block_inscription_numbers (ordinals#2667) * Hide protocol inscriptions (ordinals#2674) * Don't color links in headers (ordinals#2678) * Add inscription charms (ordinals#2681) * Group rune server tests (ordinals#2685) * Add inscription compression (ordinals#1713) * Fix media table formatting (ordinals#2686) * Update schema version for charms (ordinals#2687) * Fix unbound outpoint server error (ordinals#2479) * Add binary media type (ordinals#2671) * Clean up install.sh (ordinals#2669) * Add /collections Page (ordinals#2561) * Preview font inscriptions (ordinals#2692) * Only load used language highlight module in code preview (ordinals#2696) * Only try to create the database if it wasn't found (ordinals#2703) * Move postage into batch file (ordinals#2705) * Add destination field to batch (ordinals#2701) * Use sequence numbers database keys (ordinals#2664) * Update redb to 1.4.0 (ordinals#2714) * Refactor inscriptions paginations (ordinals#2715) * Display table stats in `ord index info` (ordinals#2711) * Use redb's recovery callback API (ordinals#2584) * Allow setting CSP origin (ordinals#2708) * Remove default file path from `ord index export --tsv` (ordinals#2717) * Use icons in nav bar (ordinals#2722) * Add Debian packaging instructions (ordinals#2725) * Add Homebrew install instructions to readme (ordinals#2726) * Add sat recursive endpoints with index and pagination (ordinals#2680) * Only accept sat number in recursive endpoint (ordinals#2732) * Fix typo in docs/src/inscriptions/metadata.md (ordinals#2731) * Add docs for metadata recursive endpoint (ordinals#2734) * Remove `RUNE` from <h1> on /rune (ordinals#2728) * Add /r/children recursive endpoint (ordinals#2431) * Add docs and examples for sat recursive endpoint (ordinals#2735) * Ignore flaky test (ordinals#2742) * Update docs to include all fields, including content-encoding (ordinals#2740) * Add docs for child recursive endpoint (ordinals#2743) * Hide JSON and .btc (ordinals#2744) * Release 0.12.0 (ordinals#2746) * Hide all text (ordinals#2753) * Add batch to preview command (ordinals#2752) * Add stuttering curse (ordinals#2745) * Batch inscribe on same sat (ordinals#2749) * Allow setting the sat to inscribe (ordinals#2765) * Select further away coins which meet target (ordinals#2724) * Fix typos (ordinals#2768) * Add ability to specify sat to batch inscribe (ordinals#2770) * Add commands to etch and list runes (ordinals#2544) * Set CSP origin in deploy script (ordinals#2764) Co-authored-by: raph <raphjaph@protonmail.com> * Add `public` to /content Cache-Control headers (ordinals#2773) * Release 0.12.1 (ordinals#2776) * Bless cursed inscriptions after Jubilee height (ordinals#2656) * Hide /content/<INSCRIPTION_ID> HTML inscriptions (ordinals#2778) * Release 0.12.2 (ordinals#2780) * fix(test): error test from version 0.12.2 --------- Co-authored-by: raph <raphjaph@protonmail.com> Co-authored-by: duttydeedz <142775511+duttydeedz@users.noreply.github.com> Co-authored-by: Casey Rodarmor <casey@rodarmor.com> Co-authored-by: liam <31192478+terror@users.noreply.github.com> Co-authored-by: Eloc <42568538+elocremarc@users.noreply.github.com> Co-authored-by: Julian Eager <eagr@tutanota.com> Co-authored-by: ordinally <11798624+veryordinally@users.noreply.github.com> Co-authored-by: Christopher Berner <me@cberner.com> Co-authored-by: Rijndael <115941166+rot13maxi@users.noreply.github.com> Co-authored-by: vuittont60 <81072379+vuittont60@users.noreply.github.com> Co-authored-by: gmart7t2 <49558347+gmart7t2@users.noreply.github.com> Co-authored-by: xiaolou86 <20718693+xiaolou86@users.noreply.github.com>

terror and others added 2 commits February 12, 2023 21:21

Add initial work

28bfb57

Merge branch 'master' into brotli

7d0a9a8

Remove todo comment

f1f2055

terror and others added 10 commits February 12, 2023 23:34

Add a bool for compress

f8fe2bc

Get it to compile

b1d21d1

Remove comment

5ca35a9

Merge branch 'master' into brotli

249ea63

Test the compression code path

3f2b6d7

Merge branch 'master' into brotli

767565a

Merge branch 'master' into brotli

897f217

Merge branch 'master' into brotli

25efeca

Fix tests

4a6fd01

Remove comment

2d58bb8

terror commented Feb 24, 2023

View reviewed changes

src/inscription.rs Outdated Show resolved Hide resolved

terror commented Feb 24, 2023

View reviewed changes

src/inscription.rs Outdated Show resolved Hide resolved

terror added 4 commits February 24, 2023 18:34

Merge branch 'master' into brotli

ea52fc8

Compress everything

24b639b

Format

18c68b8

Use odd value for content encoding

1d93d3f

This was referenced Mar 8, 2023

metadata string added for inscribe file #1899

Closed

Add compression via content-encoding #891

Closed

tyjvazum mentioned this pull request May 13, 2023

Feature: Content Compression tyjvazum/arb#2

Closed

1 task

staccDOTsol mentioned this pull request Oct 26, 2023

Features/brotlimagick #2589

Closed

terror and others added 4 commits November 13, 2023 15:35

Merge branch 'master' into brotli

7175437

Fix tests

65d633b

Format

1d6957b

Merge branch 'master' into brotli

f345a06

terror added 3 commits November 14, 2023 13:59

Remove/rearrange tests

b15a150

Add compression integration tests

d72f99c

Fix preview bs

ac5b57b

terror marked this pull request as ready for review November 14, 2023 22:45

casey requested changes Nov 14, 2023

View reviewed changes

terror added 2 commits November 14, 2023 18:25

Fix nits

aca9720

Is acceptable ?!

5f0a2c4

casey requested changes Nov 14, 2023

View reviewed changes

src/subcommand/server.rs Outdated Show resolved Hide resolved

src/subcommand/server/accept_encoding.rs Outdated Show resolved Hide resolved

src/subcommand/server/accept_encoding.rs Outdated Show resolved Hide resolved

terror added 3 commits November 14, 2023 18:34

Reference self

e49f415

To string it

4190656

More nits

02f80ad

terror and others added 12 commits November 16, 2023 13:19

Merge branch 'master' into brotli

6ac111d

Switch to using dropbox brotli impl

b02b554

Fix preview tests

ef2d85c

Fix test

5ed2525

Merge branch 'master' into brotli

be38a2d

Fix import

86d4416

Ensure we get proper decompressed value

1549caa

Use .as_slice

00e1270

Format table

5f3c3ac

Pass header to is_acceptable

4c2c13a

Remove useless tests

5faaf54

Merge branch 'master' into brotli

d0947e5

casey approved these changes Nov 16, 2023

View reviewed changes

casey enabled auto-merge (squash) November 16, 2023 20:22

casey merged commit ab047f7 into ordinals:master Nov 16, 2023
6 checks passed

terror deleted the brotli branch November 16, 2023 20:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add brotli compression for text-based inscriptions #1713

Add brotli compression for text-based inscriptions #1713

terror commented Feb 13, 2023 •

edited

Loading

casey commented Feb 13, 2023

Psifour commented Feb 13, 2023

nammaki commented Mar 23, 2023

casey left a comment

terror commented Nov 15, 2023 •

edited

Loading

casey left a comment

Add brotli compression for text-based inscriptions #1713

Add brotli compression for text-based inscriptions #1713

Conversation

terror commented Feb 13, 2023 • edited Loading

casey commented Feb 13, 2023

Psifour commented Feb 13, 2023

nammaki commented Mar 23, 2023

casey left a comment

Choose a reason for hiding this comment

terror commented Nov 15, 2023 • edited Loading

casey left a comment

Choose a reason for hiding this comment

terror commented Feb 13, 2023 •

edited

Loading

terror commented Nov 15, 2023 •

edited

Loading