Make clickhouse binary a self extracting executable.

[alexey-milovidov]

Use case

1. More quick downloads but without the need of unpacking or installing any separate tools.
2. Include odbc-bridge and library-bridge in single-binary ClickHouse downloads despite the fact that they are separate binaries.
3. Maybe the size will be ok to always include debug info, even in .deb/.rpm packages.
4. It will also speed up cold start in the cloud.
5. clickhouse-local build small enough to fit in a Lambda function #29378

Describe the solution you'd like

Compile a tool that can compress and decompress with ZSTD. It can use the official zstd framing format but not necessarily. It should support compression by blocks of size around 10..100 MiB (to avoid too high memory usage) and checksums. It should not depend on glibc version, and even better if it will be statically linked. Maybe two tools - one for compression and another for decompression.

Post-build rule will compress clickhouse (and possibly clickhouse-odbc-bridge and clickhouse-jdbc-bridge) binary into blob and then include it into the decompressor with llvm-objcopy as a custom ELF section (alternatively - maybe we can concatenate it after the decompressor binary).

Decompressor should check the free space in the current directory, decompress the content into temporary file with a similar name (like clickhouse.tmp), perform fsync, rename the current binary to a file with similar name (like .compressed), rename the decompressed binary to the original name, delete the compressed binary and run the decompressed binary with the same arguments.

If clickhouse-odbc-bridge and clickhouse-odbc-bridge binaries are present, they should be extracted into current directory and clickhouse install script should check if they are present.

Describe alternatives you've considered

There are existing specialized tools for compressing executables (like UPX). But zstd will do it better.

Additional context

A shortcut for clickhouse install in the decompressor can be implemented to avoid extra file copies (first into current directory then to the install destination like /usr/bin).

It makes sense to parallelize decompression.

[azat]

@yakov-olkhovskiy AFAICS this issue is still not resolved, since #38447 only generates such binary, but it is not used anywhere. Am I missing something? Or the initial plan had been changed?

[yakov-olkhovskiy]

@azat
oh, missed PR was marked as closing this issue
reopening
I will change deployment script tomorrow since it will depend on having this executable ready

[yakov-olkhovskiy]

not finished yet

[nikitamikhaylov]

Another idea is just to compress debug-symbols. Found an interesting article. It says that the best option is to use dwz (DWARF table optimiser) + llvm-objcopy --compress-debug-sections. For ClickHouse this approach reduces around 50% binary. I just downloaded 2.1G release binary with symbols from CI. After using dwz it became 1.9G and when I compressed debug-symbols it because 883Mb which more or less Ok. Note, that stripped binary is around 450Mb. Here is bloaty output for clickhouse

ubuntu@ip-10-1-13-116:~/trash$ bloaty clickhouse
    FILE SIZE        VM SIZE
 --------------  --------------
  30.2%   640Mi   0.0%       0    .debug_info
  20.1%   426Mi   0.0%       0    .debug_loc
  13.6%   287Mi  63.6%   287Mi    .text
  11.8%   250Mi   0.0%       0    .debug_str
   5.5%   116Mi   0.0%       0    .debug_ranges
   5.3%   113Mi   0.0%       0    .debug_line
   4.5%  94.6Mi   0.0%       0    .strtab
   2.5%  52.8Mi  11.7%  52.8Mi    .rodata
   2.4%  51.8Mi  11.4%  51.8Mi    .dynstr
   1.3%  27.0Mi   6.0%  27.0Mi    .eh_frame
   0.9%  19.7Mi   0.0%       0    .symtab
   0.6%  11.8Mi   0.0%       0    .debug_abbrev
   0.5%  9.78Mi   2.2%  9.78Mi    .dynsym
   0.4%  9.54Mi   2.1%  9.54Mi    .gcc_except_table
   0.0%       0   0.9%  4.25Mi    .bss
   0.2%  3.99Mi   0.9%  3.99Mi    .eh_frame_hdr
   0.1%  3.03Mi   0.7%  3.03Mi    .gnu.hash
   0.1%  1.69Mi   0.4%  1.68Mi    .data
   0.0%   834Ki   0.2%   834Ki    .gnu.version
   0.0%  53.7Ki   0.0%  36.1Ki    [25 Others]
   0.0%       0   0.0%  25.1Ki    .tbss
 100.0%  2.07Gi 100.0%   452Mi    TOTAL

and for binary with compressed debug symbols

ubuntu@ip-10-1-13-116:~/trash$ bloaty clickhouse.dwz.compressed
    FILE SIZE        VM SIZE
 --------------  --------------
  32.6%   287Mi  63.6%   287Mi    .text
  22.3%   196Mi   0.0%       0    .debug_info
  10.7%  94.6Mi   0.0%       0    .strtab
   6.0%  52.9Mi   0.0%       0    .debug_loc
   6.0%  52.8Mi  11.7%  52.8Mi    .rodata
   5.9%  51.8Mi  11.4%  51.8Mi    .dynstr
   3.4%  30.1Mi   0.0%       0    .debug_str
   3.1%  27.0Mi   6.0%  27.0Mi    .eh_frame
   2.8%  24.5Mi   0.0%       0    .debug_line
   2.2%  19.7Mi   0.0%       0    .symtab
   1.5%  13.4Mi   0.0%       0    .debug_ranges
   1.1%  9.78Mi   2.2%  9.78Mi    .dynsym
   1.1%  9.54Mi   2.1%  9.54Mi    .gcc_except_table
   0.0%       0   0.9%  4.25Mi    .bss
   0.5%  3.99Mi   0.9%  3.99Mi    .eh_frame_hdr
   0.3%  3.03Mi   0.7%  3.03Mi    .gnu.hash
   0.2%  1.80Mi   0.0%       0    .debug_abbrev
   0.2%  1.69Mi   0.4%  1.68Mi    .data
   0.1%   834Ki   0.2%   834Ki    .gnu.version
   0.0%  52.8Ki   0.0%  36.1Ki    [25 Others]
   0.0%       0   0.0%  25.1Ki    .tbss
 100.0%   882Mi 100.0%   452Mi    TOTAL

We can then even use upx to reduce the binary size more dramatically. But for me it did't work out the box and after executing this the upx'ed ClickHouse binary started to produce Code: 464. DB::Exception: The ELF is truncated (section header points after end of file). (CANNOT_PARSE_ELF), Stack trace (when copying this message, always include the lines below): error. Didn't dig deeper into it. But this is some output of bloaty and file.

ubuntu@ip-10-1-13-116:~/trash$ bloaty clickhouse.stripped
    FILE SIZE        VM SIZE
 --------------  --------------
   0.0%       0  74.4%   336Mi    [LOAD #1 [RW]]
 100.0%   115Mi  25.6%   115Mi    [LOAD #0 [RX]]
   0.0%  1.09Ki   0.0%       0    [Unmapped]
 100.0%   115Mi 100.0%   452Mi    TOTAL
ubuntu@ip-10-1-13-116:~/trash$ file clickhouse.stripped
clickhouse.stripped: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, no section header

This is much more simpler than writing our own compressor / decompressor and dealing with tons of CMake code and arguments escaping. And it will simplify debugging a lot without installing additional dbg package.
We can even put upx'ed binary to Alpine-based imaged and binary with compressed debug-info to Ubuntu-based images. And name them clickhouse-server and clickhouse-server-slim for example. WDYT @alexey-milovidov ?

[azat]

@nikitamikhaylov There was already an attempt to do so, by seems that internal DWARF interpreter cannot handle compressed debug symbols - #18952 (comment)

And adding this support may make it slower and this will increase random latencies when the stack will be unwinded from non-cached locations, while extracting it each time at start may take some extra time, which also not a good way to go for me.

So, not sure that it will be better, plus right now, the major problem I guess is the artifacts size not the official packages size, since later does not created too often, unlike artifacts.

[nikitamikhaylov]

@azat I saw problem with Dwarf parser only when using upx. When just compress debug symbols, everything works correctly but seems these symbols are not being used... For example when I just send 11 signal to running server

2022.08.04 12:38:49.480834 [ 710212 ] {} <Trace> BaseDaemon: Received signal 11
2022.08.04 12:38:49.481026 [ 710509 ] {} <Fatal> BaseDaemon: ########################################
2022.08.04 12:38:49.481068 [ 710509 ] {} <Fatal> BaseDaemon: (version 22.8.1.825 (official build), build id: 7E15CA8E88D65698) (from thread 710211) (no query) Received signal Segmentation fault (11)
2022.08.04 12:38:49.481083 [ 710509 ] {} <Fatal> BaseDaemon:  Access: read. Unknown si_code.
2022.08.04 12:38:49.481098 [ 710509 ] {} <Fatal> BaseDaemon: Stack trace: 0x7f80b570b376 0xa579bdb 0xa3c790e 0x19130646 0xa3b26de 0xa3afaa8 0xa311519 0x7f80b552e083 0xa0d262e
2022.08.04 12:38:49.481120 [ 710509 ] {} <Fatal> BaseDaemon: 2. pthread_cond_wait in ?
2022.08.04 12:38:49.481144 [ 710509 ] {} <Fatal> BaseDaemon: 3. BaseDaemon::waitForTerminationRequest() in /home/ubuntu/trash/clickhouse.dwz.compressed
2022.08.04 12:38:49.481162 [ 710509 ] {} <Fatal> BaseDaemon: 4. DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) in /home/ubuntu/trash/clickhouse.dwz.compressed
2022.08.04 12:38:49.481177 [ 710509 ] {} <Fatal> BaseDaemon: 5. Poco::Util::Application::run() in /home/ubuntu/trash/clickhouse.dwz.compressed
2022.08.04 12:38:49.481189 [ 710509 ] {} <Fatal> BaseDaemon: 6. DB::Server::run() in /home/ubuntu/trash/clickhouse.dwz.compressed
2022.08.04 12:38:49.481200 [ 710509 ] {} <Fatal> BaseDaemon: 7. mainEntryClickHouseServer(int, char**) in /home/ubuntu/trash/clickhouse.dwz.compressed
2022.08.04 12:38:49.481213 [ 710509 ] {} <Fatal> BaseDaemon: 8. main in /home/ubuntu/trash/clickhouse.dwz.compressed
2022.08.04 12:38:49.481222 [ 710509 ] {} <Fatal> BaseDaemon: 9. __libc_start_main in ?
2022.08.04 12:38:49.481236 [ 710509 ] {} <Fatal> BaseDaemon: 10. _start in /home/ubuntu/trash/clickhouse.dwz.compressed
2022.08.04 12:38:49.634426 [ 710509 ] {} <Fatal> BaseDaemon: Integrity check of the executable successfully passed (checksum: FEFEA47369F0FA0C30683975BF8DC041)

And add2line started to work extremely slow...