Error: Password required for user

[ipolevoy]

Hi, there. We ran into an issue configuring two servers with replication. Our cluster is:

• 1 Shard
• 2 Replicas

Replication works from node 1 to node 2. However, it does not work from node 2 to node 1. Our configuration is identical on both nodes with users, permissions, config, etc.

Here is the stack trace:

2017.03.20 17:28:09.615767 [ 3 ] campaign_events.Distributed.DirectoryMonitor: Code: 194, e.displayText() = DB::Exception: Received from 10.2.1.72:9000. DB::Exception: Password required for user user1. Stack trace:

0. clickhouse-server(StackTrace::StackTrace()+0x16) [0x12bed96]
1. clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x1f) [0x107267f]
2. clickhouse-server(DB::Users::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>
, std::allocator<char> > const&, Poco::Net::IPAddress const&) const::{lambda()#1}::operator()() const+0x5d) [0x12917cd]
3. clickhouse-server(DB::Users::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>
, std::allocator<char> > const&, Poco::Net::IPAddress const&) const+0x2ac) [0x129496c]
4. clickhouse-server(DB::Context::setUser(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits
<char>, std::allocator<char> > const&, Poco::Net::SocketAddress const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x6c) [0x1286ccc]
5. clickhouse-server(DB::TCPHandler::receiveHello()+0x80e) [0x108428e]
6. clickhouse-server(DB::TCPHandler::runImpl()+0x1cc) [0x10872bc]
7. clickhouse-server(DB::TCPHandler::run()+0x2b) [0x1088d7b]
8. clickhouse-server(Poco::Net::TCPServerConnection::start()+0xf) [0x325573f]
9. clickhouse-server(Poco::Net::TCPServerDispatcher::run()+0x13b) [0x325bb7b]
10. clickhouse-server(Poco::PooledThread::run()+0xb7) [0x34c7807]
11. clickhouse-server(Poco::ThreadImpl::runnableEntry(void*)+0xa5) [0x3493845]
12. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7facc0fd06ba]
13. /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7facc05f182d]

The user has the same password configured on both servers the same way.

Any help is much appreciated!

[alexey-milovidov]

Hi.

This is possible, if some data was inserted to Distributed table before password was specified in cluster configuration. The data inserted to Distributed tabl is written asynchronously - first it is written to filesystem on local server (look at data directory for table /path_to_clickhouse/data/db/table/), and then sent to remote server. Full address of remote server along with user name and password are encoded in directory name.

So, if you have written some data to Distributed table, when password was incorrect in cluster configuration, that incorrect password will be saved and used in subsequent tries to send the data.
Solution: manually rename directory inside /path_to_clickhouse/data/db/table/.

[ipolevoy]

@alexey-milovidov we have solved this a month ago by reinstalling everything, but still nice to know - thanks

[alexey-milovidov]

Sorry for the delay.

[ipolevoy]

@alexey-milovidov no need to be sorry, this is not a paid service :)

[abfisher0417]

This worked like a charm. I will also add that clickhouse-server needs to be restarted after manually renaming directories to include the password; otherwise, you'll see something like: Distributed.DirectoryMonitor: Poco::Exception. Code: 1000 File not found in the logs.