Add self extracting executable

[FArthur-cmd]

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Add self extracting executable #34755

[alexey-milovidov]

But why cannot we run it directly, without bash?
ClickHouse should not require shell interpreter to run.

[FArthur-cmd]

We have to execute two programs: replace current binary with other file (clickhouse.decompressed) and execute it. I found solution only with bash. I'm open to ideas.

[alexey-milovidov]

Use rename function?

[FArthur-cmd]

From man: If newpath already exists, it will be atomically replaced, so that there is no point at which another process attempting to access newpath will find it missing. However, there will probably be a window in which both oldpath and newpath refer to the file being renamed.
On practice it renames successfully, but execve tries to execute decompressor instead other file, and fails with Text file busy. But next command to this binary will be successful.

[alexey-milovidov]

Btw, compressor can be written in C++ with exceptions, RAII, etc...
Only decompressor is intended to be lightweight.

[FArthur-cmd]

I made it to make the compressor and decompressor look the same. I can change it, but this version looks good to me.

[alexey-milovidov]

Ok.

[alexey-milovidov]

@yakov-olkhovskiy one month has passed, I will reassign.

[yakov-olkhovskiy]

@alexey-milovidov missed this one doing other projects... can do next week

[nikitamikhaylov]

@yakov-olkhovskiy Did you test it with ClickHouse directly? It would be nice to add it by default to our containers

[yakov-olkhovskiy]

@nikitamikhaylov no, I didn't - only tested with some helloworld executable to check arguments and environment transfer

[antonio2368]

Before reviewing the code I noticed it's written more in a C-style.
What do you think about rewriting it first in a more C++-style (e.g. replacing new[], delete[] with std::vector)?

EDIT: now I see that a lot of C++ code was removed on purpose.

[nikitamikhaylov]

It would be also nice to forward arguments to a binary when we do decompression and execute it. From quick look into code I don't understand whether we do it or not

[FArthur-cmd]

@nikitamikhaylov If I understand you correctly

ClickHouse/utils/self-extr-exec/decompressor.cpp

Line 368 in 12e7770

execvp(bash, newargv);

[yakov-olkhovskiy]

@nikitamikhaylov we do

ClickHouse/utils/self-extr-exec/decompressor.cpp

Lines 268 to 297 in 12e7770

	/// Set command to `mv filename.decompressed filename && filename agrs...`
	void fillCommand(char command[], int argc, char * argv[], size_t length)
	{
	memset(command, '\0', 3 + strlen(argv[0]) + 14 + strlen(argv[0]) + 4 + strlen(argv[0]) + length + argc);
	/// position in command
	size_t shift = 0;
	/// Support variables to create command
	char mv[] = "mv ";
	char decompressed[] = ".decompressed ";
	char add_command[] = " && ";
	char space[] = " ";
	fill(command, mv, 3, shift);
	fill(command, argv[0], strlen(argv[0]), shift);
	fill(command, decompressed, 14, shift);
	fill(command, argv[0], strlen(argv[0]), shift);
	fill(command, add_command, 4, shift);
	fill(command, argv[0], strlen(argv[0]), shift);
	fill(command, space, 1, shift);
	/// forward all arguments
	for (int i = 1; i < argc; ++i)
	{
	fill(command, argv[i], strlen(argv[i]), shift);
	if (i != argc - 1)
	fill(command, space, 1, shift);
	}
	}

[yakov-olkhovskiy]

@mergify update

[mergify]

update

✅ Branch has been successfully updated

[alexey-milovidov]

Why do we run it with sh -c?
Why is shell needed?

It makes everything more complicated, and proper argument forwarding is nearly impossible.

I think ClickHouse should run under chroot with no shell.

[yakov-olkhovskiy]

as I remember the reason for using sh was that attempt to run just clickhouse gives "text file is busy" error.
And what about chroot here? how it helps?

[alexey-milovidov]

You can workaround the "text file busy" error by renaming the old executable to a temporary file first.
Chroot will not help, it is just a motivation to not use sh.

[yakov-olkhovskiy]

will try

[yakov-olkhovskiy]

@alexey-milovidov doesn't work either. Algorithm is next:

system("mv " + argv[0] + " " + argv[0] + ".tmp");
system("mv " + argv[0] + ".decompressed " + argv[0]);
system("rm " + argv[0] + ".tmp"); // optional, result is the same without it
execvp(argv[0], argv);

result the same - last execvp gives "Text file busy" error

[yakov-olkhovskiy]

@alexey-milovidov BUT replacing mv clickhouse.decompressed clickhouse by cp clickhouse.decompressed clickhouse seems to work! :)

[yakov-olkhovskiy]

@alexey-milovidov #39763