http 2.0.0

[nabijaczleweli]

There's a potential release issue in that fuck knows how well cargo will take to our overrides. And that can only be tested live. So I'm gonna be testing it live in a bit.

BREAKING:

• id=s of files (#-anchors) in directory listings changed to be the filename but tr " _, rather than filename but tr . _ (and "s unescaped!)
• id=parent_dir changed to id=..

Fixed:

• Blockdev size detexion on musl/*BSD/&c. (Build fails on Android/Termux #155) (FreeBSD cycle detected when const-evaluating #157) (Support BSDs by special-casing Linux and Android specifically #158)
• Truncated-at-100-entries directory uploads on Chromium
• MacOS marking all files as WebDAV executable
• Potentially hitting NAME_MAX when deciding on temporary directory
• Potentially using path of removed file for cached-as-incompressible files
• Temporary file for uploads not being removed
• All CPUs going 100% for a few seconds on Ctrl-C on Win32 sometimes
• /'s index with -wd first directory having full-width instead of square icons
• Max (11) brotli encoding strength reduced to 9 (11 takes 50s to encode 11M!)
• PROPFIND of <Win32LastAccessTime> returning <Win32FileAttributes> with <Win32LastAccessTime>'s data
• -m only taking UTF-8 data even though it's for paths
• i Closed range requests reading the whole response body into a buffer
• PROPFIND Depth: infinity panicking when encountering non-directories
• Don't require the current extension to be UTF-8 when encoding a file
• Localising stamps in 1000-file directory on Android 14 Browser.apk (Chrome): 8+s -> ~200ms (solution by @the-eater) (https://cijber.social/@eater/112506297971225464)
• Double-click-to-select text in web UI rename box
• " unescaped in listing href=s, making them unnavigable
• If encoded file was removed from temp dir, re-encode it instead of panicking
• Clicking on the Manage column of Create directory submitting an empty MKCOL
• h hyper RUSTSEC-2021-0078: potential desync with Content-Length: +123 (Multiple dependencies with known security vulnerabilities #140)
• h hyper RUSTSEC-2021-0079: potential desync with Transfer-Encoding: chunked (Multiple dependencies with known security vulnerabilities #140)
• h traitobject RUSTSEC-2020-0027: reliance on fat pointer representation (Multiple dependencies with known security vulnerabilities #140)
• background-color in directory listing adapted to potential dark theme
• Files called file_upload and new_directory potentially interfering with directory listing controls
• Add id=.. in mobile listing

Changed:

• Double-iteration over directory uploads removed
• Don't-try-to-encode extension list updated
• ETag generator is also used to cache file hashes (=> file not read twice)
• i Always fully 1M-buffer file I/O (used to be 8k)
• Also remove the top-level temporary directory (but ignore error)
• h Start threads on demand, up to the limit, then decay on inactivity (wait 5threadnum seconds before exit), but keep 1, rather than always having nproc8
• Re-compressed favicon.png (1460B) now used as the favicon inlined in indices instead of a 15k(!+base64) ICO
• h Never-used bzip2 encoding removed (+ asking for zstd encoding is alloc-free)
• PUT symlink isolation is the same as with GET's (returns 404, doesn't read)
• Useless HTML attributes in generated responses removed
• Rename UI suggested filename no longer includes trailing / for directories
• Replaced <meta name="application-name" content="http"> & <meta name="author" content="http developers"> with <meta name="generator" content="http ${VERSION}">
• Mobile directory listings simplified and reduced by 17% in size
• Desktop directory listings only allow tabbing to /one/ link per row
• Desktop directory listings have "Create directory" colspan=3

New:

• Return ETag and honour If-None-Match/If-Modified-Since for GETs
• Live read-out of outstanding requests when uploading from generated index
• Invent X-Last-Modified: millis-since-epoch header to set mtime on PUT files
• Accept X-OC-MTime: seconds-since-epoch header (Android Totalcmd-WebDAV)
• Actually handle PROPPATCH at all, and allow changing:
  • Win32{Creation,Last{Access,Modified}}Time (Windows client)
  • executable (davfs2)
• i Limits on size of encoded files with --encoded-{filesystem,generated} 20M
• i Limits on max age of encoded files with --encoded-prune 1d
• -Q/--quiet-time to remove [%F %T] prefix in log
• If PUTted file doesn't already exist, copy it directly from the request body
• Accept (ignore) renaming to current name in web UI
• Return 503 Service Unavailable with an intelligible error instead of panicking on PUT errors
• Display these in the web UI
• "confirm" (green tick) icon
• Linked-to files in directory listing (like #asdf) highlighted

Internal:

• Needless Borrow::borrow() call removed (report: roz@rozbrajacz.futbol)
• Literalise mime types instead of parsing
• Don't check if temporary directory exists before trying to delete
• Useless lazy_static and regex dependencies killed
• Run-time parsing of do-not-encode list replaced with a codegenned function (also kills unicase)
• Assets and indices are pre-parsed in build.rs before embedding (also kills base64 and lazysort) (+ initial whitespace trimmed from HTMLs)
• h Cache allowed methods instead of reallocating every time
• Vendored: iron 0.6.1 (with typemap, plugin, and log deps deleted)
• i Removed automatic annoying Content-Type muddling in iron
• Static Ctrl-C handler Condvar
• Don't allocate URL path if not necessary, and escape it at most once
• Vendored: hyper 0.10.16 (with client component, some server body types, and log and env_logger dependencies deleted)
• h Hash encoded-data caches by Encoding(Type) directly instead of String repr; this also makes us stop distinguishing x-gzip from gzip
• h Don't require reallocating Server header contents
• h Don't require allocating RangeUnits in Accept-Ranges header
• h Use SmallVec for most common case of 1 Content-Encoding
• h Don't reallocate custom --header names
• h Don't allocate for COPY/MKCOL/MOVE/PROPFIND/PROPPATCH/LOCK WebDAV requests
• Don't clone headers for TRACE
• h language-tags dependency removed
• .. + 1 => ..=
• os-str-generic dependency removed
• Recompress rename icon (206 -> 192 bytes)
• Vendored: rfsapi 0.2.0
• ir Header parsing takes Cow<'static, [u8]> instead of Vec<u8>
• h Use VecMap as backing store for Header's PtrMap instead of HashMap
• Free unused bits of Options
• Don't allocate file length or 404 description as string
• Assess file size once, from opened file, for opened ranges
• Don't reallocate Range header value
• Use the same buffer for recursive PROPFINDs
• Produce address indents with {:w$} instead of Spaces(w) writing ' ' w times
• Don't allocate in AddressWriter
• Don't allocate human-readable size
• date.js replaced with standard JS Date manipulation
• All document onload => DOMContentLoaded
• In a similar vein, move the click event listeners to onclick (perf)
• Don't reallocate each filename twice to replace & and < in index
• Re-save embedded-in-EXE favicon.ico (15k -> 2k)
• Flatten Error to just be a String, instead of three with auto-ingification
• Detect EADDRINUSE explicitly when trying new ports, rather than guessing from the I/O error stringification
• Ignore RUSTSEC-2021-0145: we don't use a custom allocator
• Ignore cargo-audit's "unmaintained" warnings
• Ignore RUSTSEC-2020-0071: we do not setenv ever
• Ignore RUSTSEC-2022-0022: our version of hyper never uses uninitialized() (Multiple dependencies with known security vulnerabilities #140)
• Desktop directory listing simplified (inlined style, useless <div>)

i: affects vendored iron; h: affects vendored hyper; r: affects vendored rfsapi

Closes: #140

[nabijaczleweli]

Real diffstat (not including vendored deps):

$ git diff --stat v1.13.2  build* http.md assets src
 assets/adjust_tz.js                  |   17 ++-
 assets/date.js                       |   10 --
 assets/directory_listing.html        |   42 +++----
 assets/directory_listing_mobile.html |   69 +++--------
 assets/encoding_blacklist            |   95 ++++++++------
 assets/error.html                    |    7 +-
 assets/favicon.ico                   |  Bin 15086 -> 1905 bytes
 assets/favicon.png                   |  Bin 4579 -> 1460 bytes
 assets/icons/LICENSE                 |    4 +-
 assets/icons/confirm.gif             |  Bin 0 -> 102 bytes
 assets/icons/confirm.png             |  Bin 568 -> 0 bytes
 assets/icons/rename.gif              |  Bin 0 -> 192 bytes
 assets/icons/rename.png              |  Bin 206 -> 0 bytes
 assets/manage.js                     |  100 +++++++--------
 assets/manage_desktop.js             |   54 ++++----
 assets/manage_mobile.js              |   66 +++++-----
 assets/upload.js                     |   57 +++++----
 build-ioctl.c                        |   51 ++++++++
 build.rs                             |  209 +++++++++++++++++++++++++------
 http.md                              |   44 +++++++
 src/error.rs                         |   42 -------
 src/main.rs                          |   82 +++++--------
 src/ops/bandwidth.rs                 |   12 +-
 src/ops/mod.rs                       | 1232 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------------------------------------------------------------------------------
 src/ops/prune.rs                     |  149 ++++++++++++++++++++++
 src/ops/webdav.rs                    |  179 +++++++++++++++++++--------
 src/options.rs                       |  129 +++++++++++++++----
 src/util/content_encoding.rs         |   83 +++++--------
 src/util/mod.rs                      |  363 +++++++++++++++++++++++++++++-------------------------
 src/util/os/non_windows.rs           |   96 ++++++++++++++-
 src/util/os/non_windows_non_macos.rs |   43 ++-----
 src/util/os/windows.rs               |   70 ++++++++++-
 src/util/os/windows_macos.rs         |    6 -
 src/util/webdav.rs                   |   39 +++---
 34 files changed, 2046 insertions(+), 1304 deletions(-)

[nabijaczleweli]

aaaand here's the issue:

P:\Rust\http>cargo publish
warning: `D:\Users\nabijaczleweli\.cargo\config` is deprecated in favor of `config.toml`
note: if you need to support cargo 1.38 or earlier, you can symlink `config` to `config.toml`
    Updating crates.io index
error: all dependencies must have a version specified when publishing.
dependency `iron` does not specify a version
Note: The published dependency will use the version from crates.io,
the `path` specification will be removed from the dependency declaration.

god.

[nabijaczleweli]

Yeah okay this is rust-lang/cargo#1565
and https://users.rust-lang.org/t/is-it-possible-to-publish-crates-with-path-specified/91497

And this would involve publishing three different pieces of shit (the vendored deps), then fucking up our dependencies, then publishing http directly. On every release.

cargo resists being useful at every step of the way

I am instead opting to re-route our release model as

• crates.io https 2.0.0 fails to build and says "run cargo install --git https://github.com/thecoshman/http"
• we prescribe installing from git

[nabijaczleweli]

he he he

but @thecoshman this does need to be approved ASAP now (for me to ff-merge manually) because git deps pull from the default branch

[nabijaczleweli]

Actually the best course of action would be to unprotect master. Because I could do releases (just off develop) previously but now I can't anymore, because the release channel is now master, so if you disappear or go on a prolonged sabbatical again this leaves everyone without a release channel.

Rest assured that many versions of this were developed and thoroughly tested by serving all *.nabijaczleweli.xyz services and https://hinfo.network.

[thecoshman]

Have you basically baked in a good few of our dependencies? How come no more crates.io deploy?

[nabijaczleweli]

Yes, three deps are vendored now because we need to patch them (for both security advisories and performance). As for the latter: #160 (comment)

[thecoshman]

aaaand here's the issue:

P:\Rust\http>cargo publish
warning: `D:\Users\nabijaczleweli\.cargo\config` is deprecated in favor of `config.toml`
note: if you need to support cargo 1.38 or earlier, you can symlink `config` to `config.toml`
    Updating crates.io index
error: all dependencies must have a version specified when publishing.
dependency `iron` does not specify a version
Note: The published dependency will use the version from crates.io,
the `path` specification will be removed from the dependency declaration.

god.

Is this saying that it refuses to build using our baked in dependencies? That seems very stupid 😞

[nabijaczleweli]

it is very stupid yes

[thecoshman]

Whilst very sucky... is there anything to be said for forking those 'vendor' libs so we can have a lib that we can then 'just use' here?

[nabijaczleweli]

There is in that, after vendoring, i did fork them by editing them. But separating them out more is equivalent to keeping them in-tree: it would also mean uploading them all to crates.io, each time, if that's even possible (because we do need to explicitly override a dep in our Cargo.toml to get iron to pick up the right hyper (cargo is very good)).

[nabijaczleweli]

fucking great. truly A1 shit. works when building from the tree of course

[nabijaczleweli]

so either we have:

• brotli feature simd, which works when building in-tree, and is faster presumably, but doesn't work when installing via cargo install --git, or
• brotli without feature simd, which works globally but is slower presumably

cool

[nabijaczleweli]

rust-lang/cargo#14001

[nabijaczleweli]

Hotfix in 2.0.1 in #161