Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

After upgrade to 23.3.1 ClickHouse server not startup in docker #48296

Closed
oleg-savko opened this issue Mar 31, 2023 · 11 comments
Closed

After upgrade to 23.3.1 ClickHouse server not startup in docker #48296

oleg-savko opened this issue Mar 31, 2023 · 11 comments
Labels
potential bug To be reviewed by developers and confirmed/rejected.

Comments

@oleg-savko
Copy link

Clickhouse not startup in docker after upgrade from clickhouse/clickhouse-server:23.2.4 to clickhouse/clickhouse-server:23.3.1, seems some problem with detection timezone:

clickhouse-server-global    | Poco::Exception. Code: 1000, e.code() = 0, Exception: Could not determine time zone from TZ variable value: 'UTC': filesystem error: in canonical: Operation not permitted ["/usr/share/zoneinfo/"] [""], Stack trace (when copying this message, always include the lines below):
clickhouse-server-global    |
clickhouse-server-global    | 0. DateLUT::DateLUT() @ 0xe177d42 in /usr/bin/clickhouse
clickhouse-server-global    | 1. OwnPatternFormatter::formatExtended(DB::ExtendedLogMessage const&, String&) const @ 0xe3f519a in /usr/bin/clickhouse
clickhouse-server-global    | 2. DB::OwnFormattingChannel::logExtended(DB::ExtendedLogMessage const&) @ 0xe3f11f3 in /usr/bin/clickhouse
clickhouse-server-global    | 3. DB::OwnSplitChannel::logSplit(Poco::Message const&) @ 0xe3f5d99 in /usr/bin/clickhouse
clickhouse-server-global    | 4. DB::OwnSplitChannel::tryLogSplit(Poco::Message const&) @ 0xe3f5be2 in /usr/bin/clickhouse
clickhouse-server-global    | 5. DB::OwnSplitChannel::log(Poco::Message const&) @ 0xe3f5948 in /usr/bin/clickhouse
clickhouse-server-global    | 6. DB::LazyPipeFDs::tryIncreaseSize(int) @ 0xe1cce8f in /usr/bin/clickhouse
clickhouse-server-global    | 7. BaseDaemon::initializeTerminationAndSignalProcessing() @ 0xe3deeaf in /usr/bin/clickhouse
clickhouse-server-global    | 8. BaseDaemon::initialize(Poco::Util::Application&) @ 0xe3db9de in /usr/bin/clickhouse
clickhouse-server-global    | 9. DB::Server::initialize(Poco::Util::Application&) @ 0xe212a83 in /usr/bin/clickhouse
clickhouse-server-global    | 10. Poco::Util::Application::run() @ 0x1784d5f7 in /usr/bin/clickhouse
clickhouse-server-global    | 11. DB::Server::run() @ 0xe2128dc in /usr/bin/clickhouse
clickhouse-server-global    | 12. Poco::Util::ServerApplication::run(int, char**) @ 0x17860e5a in /usr/bin/clickhouse
clickhouse-server-global    | 13. mainEntryClickHouseServer(int, char**) @ 0xe20f4a6 in /usr/bin/clickhouse
clickhouse-server-global    | 14. main @ 0x899a253 in /usr/bin/clickhouse
clickhouse-server-global    | 15. ? @ 0x7fdcbca2ed90 in ?
clickhouse-server-global    | 16. __libc_start_main @ 0x7fdcbca2ee40 in ?
clickhouse-server-global    | 17. _start @ 0x7fe1aae in /usr/bin/clickhouse
clickhouse-server-global    |  (version 23.3.1.2823 (official build))
clickhouse-server-global    | Poco::Exception. Code: 1000, e.code() = 0, System exception: cannot start thread, Stack trace (when copying this message, always include the lines below):
clickhouse-server-global    |
clickhouse-server-global    | 0. Poco::ThreadImpl::startImpl(Poco::SharedPtr<Poco::Runnable, Poco::ReferenceCounter, Poco::ReleasePolicy<Poco::Runnable>>) @ 0x179be374 in /usr/bin/clickhouse
clickhouse-server-global    | 1. Poco::Thread::start(Poco::Runnable&) @ 0x179bf2cb in /usr/bin/clickhouse
clickhouse-server-global    | 2. BaseDaemon::initializeTerminationAndSignalProcessing() @ 0xe3def63 in /usr/bin/clickhouse
clickhouse-server-global    | 3. BaseDaemon::initialize(Poco::Util::Application&) @ 0xe3db9de in /usr/bin/clickhouse
clickhouse-server-global    | 4. DB::Server::initialize(Poco::Util::Application&) @ 0xe212a83 in /usr/bin/clickhouse
clickhouse-server-global    | 5. Poco::Util::Application::run() @ 0x1784d5f7 in /usr/bin/clickhouse
clickhouse-server-global    | 6. DB::Server::run() @ 0xe2128dc in /usr/bin/clickhouse
clickhouse-server-global    | 7. Poco::Util::ServerApplication::run(int, char**) @ 0x17860e5a in /usr/bin/clickhouse
clickhouse-server-global    | 8. mainEntryClickHouseServer(int, char**) @ 0xe20f4a6 in /usr/bin/clickhouse
clickhouse-server-global    | 9. main @ 0x899a253 in /usr/bin/clickhouse
clickhouse-server-global    | 10. ? @ 0x7fdcbca2ed90 in ?
clickhouse-server-global    | 11. __libc_start_main @ 0x7fdcbca2ee40 in ?
clickhouse-server-global    | 12. _start @ 0x7fe1aae in /usr/bin/clickhouse
clickhouse-server-global    |  (version 23.3.1.2823 (official build))
clickhouse-server-global    | Processing configuration file '/etc/clickhouse-server/config.xml'.
clickhouse-server-global    | Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
clickhouse-server-global    | Merging configuration file '/etc/clickhouse-server/config.d/models_config.xml'.
clickhouse-server-global    | Merging configuration file '/etc/clickhouse-server/config.d/named_connections.xml'.
clickhouse-server-global    | Include not found: clickhouse_remote_servers
clickhouse-server-global    | Include not found: clickhouse_compression
clickhouse-server-global    | Logging trace to /var/log/clickhouse-server/clickhouse-server.log
clickhouse-server-global    | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
clickhouse-server-global    | Cannot add message to the log: Sending crash reports is disabled
@oleg-savko oleg-savko added the potential bug To be reviewed by developers and confirmed/rejected. label Mar 31, 2023
@Avogar
Copy link
Member

Avogar commented Apr 3, 2023

Can you share how your upgrade process looks like? For me clickhouse/clickhouse-server:23.3.1 works fine in docker.

@oleg-savko
Copy link
Author

just change image in docker-compose.yaml
image: clickhouse/clickhouse-server:23.2.4 -> image: clickhouse/clickhouse-server:23.3.1

and restart it:
docker-compose up -d clickhouse-server

when revert image back to clickhouse/clickhouse-server:23.2.4
ch startup successfully

os version:

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.4 LTS
Release:	18.04
Codename:	bionic

docker --version
Docker version 19.03.12, build 48a66213fe

@Avogar
Copy link
Member

Avogar commented Apr 3, 2023 •
edited
Loading

Hmm. I created an instance just like yours and everything is ok for me:

ubuntu@ip-172-31-1-122:~/tmp$ docker --version
Docker version 19.03.12, build 48a66213fe
ubuntu@ip-172-31-1-122:~/tmp$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.6 LTS
Release:	18.04
Codename:	bionic
ubuntu@ip-172-31-1-122:~/tmp$ cat docker-compose.yaml
version: "3"

services:
    clickhouse:
        image: clickhouse/clickhouse-server:23.2.4
        ports:
            - 9000:9000
        volumes:
            - ./server:/var/lib/clickhouse
            - ./logs:/var/log/clickhouse-server
ubuntu@ip-172-31-1-122:~/tmp$ docker-compose up -d
Creating network "tmp_default" with the default driver
Creating tmp_clickhouse_1 ...
Creating tmp_clickhouse_1 ... done
ubuntu@ip-172-31-1-122:~/tmp$ ./clickhouse client -q "select version()"
23.2.4.12
# change version to 23.3.1
ubuntu@ip-172-31-1-122:~/tmp$ cat docker-compose.yaml
version: "3"

services:
    clickhouse:
        image: clickhouse/clickhouse-server:23.3.1
        ports:
            - 9000:9000
        volumes:
            - ./server:/var/lib/clickhouse
            - ./logs:/var/log/clickhouse-server
ubuntu@ip-172-31-1-122:~/tmp$ docker-compose up -d
Recreating tmp_clickhouse_1 ...
Recreating tmp_clickhouse_1 ... done
ubuntu@ip-172-31-1-122:~/tmp$ ./clickhouse client -q "select version()"
23.3.1.2823

Maybe you use some specific settings in configs?

@oleg-savko
Copy link
Author

After upgrading tzdata, which menthioned here, clickhouse start up successfully

#3157

@Avogar Avogar closed this as completed Apr 6, 2023
@mzealey
Copy link

mzealey commented Apr 14, 2023 •
edited
Loading

We are seeing the same issue on a centos server:

# cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
# uname -r
3.10.0-1062.12.1.el7.x86_64
# docker --version
Docker version 1.13.1, build 7d71120/1.13.1

simple script:

#!/bin/bash
IMAGE=clickhouse/clickhouse-server:23.3
docker pull $IMAGE
docker rm -f clickhouse-test
docker run -d --name clickhouse-test \
    --restart unless-stopped --ulimit nofile=262144:262144 \
        $IMAGE

produces:

Poco::Exception. Code: 1000, e.code() = 0, Exception: Could not determine time zone from TZ variable value: 'UTC': filesystem error: in canonical: Operation not permitted ["/usr/share/zoneinfo/"] [""], Stack trace (when copying this message, always include the lines below):

The host server's timezone is set up fine, running latest 22.11 and 23.2 images with the same command works perfectly, it's something that has changed in 23.3.

If I spin up the container with --entrypoint /bin/bash I see that there's some real weirdness with permissions inside the container like:

root@c76fe47803e2:/# ls -l
ls: cannot access 'bin': Operation not permitted
ls: cannot access 'boot': Operation not permitted
ls: cannot access 'dev': Operation not permitted
ls: cannot access 'etc': Operation not permitted
ls: cannot access 'home': Operation not permitted
ls: cannot access 'lib': Operation not permitted
ls: cannot access 'lib32': Operation not permitted
ls: cannot access 'lib64': Operation not permitted
ls: cannot access 'libx32': Operation not permitted
ls: cannot access 'media': Operation not permitted
ls: cannot access 'mnt': Operation not permitted
ls: cannot access 'opt': Operation not permitted
ls: cannot access 'proc': Operation not permitted
ls: cannot access 'root': Operation not permitted
ls: cannot access 'run': Operation not permitted
ls: cannot access 'sbin': Operation not permitted
ls: cannot access 'srv': Operation not permitted
ls: cannot access 'sys': Operation not permitted
ls: cannot access 'tmp': Operation not permitted
ls: cannot access 'usr': Operation not permitted
ls: cannot access 'var': Operation not permitted
ls: cannot access 'entrypoint.sh': Operation not permitted
ls: cannot access 'docker-entrypoint-initdb.d': Operation not permitted
total 0
l????????? ? ? ? ?            ? bin
d????????? ? ? ? ?            ? boot
d????????? ? ? ? ?            ? dev
d????????? ? ? ? ?            ? docker-entrypoint-initdb.d
-????????? ? ? ? ?            ? entrypoint.sh
d????????? ? ? ? ?            ? etc
d????????? ? ? ? ?            ? home
l????????? ? ? ? ?            ? lib
l????????? ? ? ? ?            ? lib32
l????????? ? ? ? ?            ? lib64
l????????? ? ? ? ?            ? libx32
d????????? ? ? ? ?            ? media
d????????? ? ? ? ?            ? mnt
d????????? ? ? ? ?            ? opt

tab-completion in the subdirectories works but ls doesn't.

Running with docker run --privileged seems to solve the issue so it looks like some sort of additional security restriction on the base 22.04 image?

@oleg-savko
Copy link
Author

Still cant to update ch to 23.3.2 from 23.2 on some servers, its cant to start:

root@es-prod-clickhouse-slave:~# docker-compose -f warehouse.yaml up
Starting clickhouse ... done
Attaching to clickhouse
clickhouse           | Processing configuration file '/etc/clickhouse-server/config.xml'.
clickhouse           | Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
clickhouse           | Merging configuration file '/etc/clickhouse-server/config.d/models_config.xml'.
clickhouse           | Merging configuration file '/etc/clickhouse-server/config.d/named_connections.xml'.
clickhouse           | Include not found: clickhouse_remote_servers
clickhouse           | Include not found: clickhouse_compression
clickhouse           | Logging trace to /var/log/clickhouse-server/clickhouse-server.log
clickhouse           | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
clickhouse           | Poco::Exception. Code: 1000, e.code() = 0, System exception: cannot start thread, Stack trace (when copying this message, always include the lines below):
clickhouse           |
clickhouse           | 0. Poco::ThreadImpl::startImpl(Poco::SharedPtr<Poco::Runnable, Poco::ReferenceCounter, Poco::ReleasePolicy<Poco::Runnable>>) @ 0x17a9fb94 in /usr/bin/clickhouse
clickhouse           | 1. Poco::Thread::start(Poco::Runnable&) @ 0x17aa0aeb in /usr/bin/clickhouse
clickhouse           | 2. BaseDaemon::initializeTerminationAndSignalProcessing() @ 0xe433703 in /usr/bin/clickhouse
clickhouse           | 3. BaseDaemon::initialize(Poco::Util::Application&) @ 0xe43017e in /usr/bin/clickhouse
clickhouse           | 4. DB::Server::initialize(Poco::Util::Application&) @ 0xe267203 in /usr/bin/clickhouse
clickhouse           | 5. Poco::Util::Application::run() @ 0x1792ee17 in /usr/bin/clickhouse
clickhouse           | 6. DB::Server::run() @ 0xe26705c in /usr/bin/clickhouse
clickhouse           | 7. Poco::Util::ServerApplication::run(int, char**) @ 0x1794267a in /usr/bin/clickhouse
clickhouse           | 8. mainEntryClickHouseServer(int, char**) @ 0xe263c26 in /usr/bin/clickhouse
clickhouse           | 9. main @ 0x89e8b13 in /usr/bin/clickhouse
clickhouse           | 10. ? @ 0x7f4125320d90 in ?
clickhouse           | 11. __libc_start_main @ 0x7f4125320e40 in ?
clickhouse           | 12. _start @ 0x803036e in /usr/bin/clickhouse
clickhouse           |  (version 23.3.2.37 (official build))

@den-crane den-crane mentioned this issue Jun 9, 2023
Closed
@UnamedRus
Copy link
Contributor

Related #49883

@Felixoid Felixoid mentioned this issue Jun 27, 2023
Merged
@Felixoid
Copy link
Member

Trying to address this in #51504 (comment)

@Felixoid
Copy link
Member

@oleg-savko can you test the latest 23.3, please?

@Fang-Li
Copy link

Fang-Li commented Dec 25, 2023

same as above like tz not permit

@Felixoid
Copy link
Member

You need to spin another issue with a detailed description. You can mention me there

@ClickHouse ClickHouse locked as resolved and limited conversation to collaborators Dec 30, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
potential bug To be reviewed by developers and confirmed/rejected.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@Felixoid @mzealey @UnamedRus @oleg-savko @Fang-Li @Avogar