Unexpected $-1 reply on set

[larroy]

(gdb) p key
$9 = "U-02266a222036de6d8f00000000"
(gdb) p value
$10 = "402266a222036de6d8f00000000\000\000\000\000\000\000\000\000\004\301\377V\262c\023&@\277\016\234\063\242|I@l&\337ls\023&@\342\344~\207\242|I@\000\200@\020\b$01266a222
000000000$0136de6d8f00000000\301\377V\262c\023&@\277\016\234\063\242|I@l&\337ls\023&@\342\344~\207\242|I@"
(gdb) down
#2 0x0000000000cf19e6 in redis::base_clientwom::WomHasher::recv_ok_reply_ (this=0x7ffff2e53c80, socket=14) at /map_int/home/larroy/devel/mmcc/frontend/external-src/redis

/redisclient.h:2263
(gdb) down
#1 0x0000000000cf5302 in redis::base_clientwom::WomHasher::recv_single_line_reply_ (this=0x7ffff2e53c80, socket=14) at /map_int/home/larroy/devel/mmcc/frontend/external-

src/redis/redisclient.h:2256
(gdb) p line
$11 = "$-1"
(gdb)

I don't know why I got $-1

[yinqiwen]

$-1 means Null multi-bulk reply. Redis defined 'set' reply format which may be a null multi-bulk reply in some case.
http://redis.io/commands/set

[larroy]

Why did it respond with a multibulk if I just did a normal set?
536│ void set(const string_type & key,
537│ const string_type & value)
538│ {
539│ int socket = get_socket(key);
540│ send_(socket, makecmd("SET") << key << value);
541├> recv_ok_reply_(socket);
542│ }

(gdb) p key
$16 = "U-02266a222036de6d8f00000000"
(gdb) p value
$17 = "402266a222036de6d8f00000000\000\000\000\000\000\000\000\000\004\301\377V\262c\023&@\277\016\234\063\242|I@l&\337ls\023&@\342\344~\207\242|I@\000\200@\020\b$01266a222
000000000$0136de6d8f00000000\301\377V\262c\023&@\277\016\234\063\242|I@l&\337ls\023&@\342\344~\207\242|I@"
(gdb)

[yinqiwen]

Is there an exist key 'U-02266a222036de6d8f00000000' in ardb which is not string?

[larroy]

shouldn't not have been, but I tried to reproduce and I didn't saw a $-1
when setting a key that already existed.

Now the database became irresponsive:

Read events are being triggered on fd with value -1. Do you have an idea of
what it means? I think an error in "accept"

209│ if (-1 == fd)
210│ {
211│ if (errno == EINTR)
212│ {
213│ continue;
214│ } else
215│ {
216├> return;
217│ }
218│ }
219│ ClientSocketChannel * ch =
GetService().NewClientSocketChannel();
220│ // if (aeCreateFileEvent(serv.GetRawEventLoop(), fd,
AE_READABLE,
221│ // Channel::IOEventCallback, ch) == AE_ERR)
222│ // {
223│ // int err = errno;
224│ // ERROR_LOG(
225│ // "Failed to add event for accepted client for
fd:%d for reason:%s", fd, strerror(err));
226│ // serv.DeleteChannel(ch);
227│ // ::close(fd);
228│ // return;
/map_int/home/larroy/devel/ardb/src/channel/socket/serversocket_channel.cpp

(gdb) bt
#0 ardb::ServerSocketChannel::OnRead (this=0x7ffff64c0780) at
channel/socket/serversocket_channel.cpp:209
#1 0x00000000004ed613 in ardb::Channel::IOEventCallback
(eventLoop=0x7ffff6432108, fd=25, clientData=0x7ffff64c0780, mask=1) at
channel/channel.cpp:50
#2 0x000000000051a1da in aeProcessEvents (eventLoop=0x7ffff6432108,
flags=3) at channel/redis/ae.c:429
#3 0x000000000051a541 in aeMain (eventLoop=0x7ffff6432108) at
channel/redis/ae.c:485
#4 0x00000000004f70b4 in ardb::ChannelService::Start (this=0x7ffff64c0500)
at channel/channel_service.cpp:323
#5 0x000000000041a0bd in ardb::ArdbServer::Start (this=0x7fffffffd5f0,
props=...) at ardb_server.cpp:3805
#6 0x000000000047d75a in main (argc=2, argv=0x7fffffffde18) at main.cpp:110
(gdb) n
(gdb) p fd
$5 = -1
(gdb) p errno
Cannot find shared library /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.15.so' in dynamic linker's load module list (gdb) n (gdb) p errno Cannot find shared library/usr/lib/debug/lib/x86_64-linux-gnu/libc-2.15.so'
in dynamic linker's load module list
(gdb) p errno
Cannot find shared library /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.15.so' in dynamic linker's load module list (gdb) p std::errno No symbol "errno" in namespace "std". (gdb) p if A syntax error in expression, nearif'.
(gdb) p fd
$6 = -1
(gdb) p m_fd
$8 = 25

lsof | grep ardb |grep "ardb-serv 25"

returns nothing.

On Wed, Feb 5, 2014 at 4:02 PM, yinqiwen notifications@github.com wrote:

Is there an exist key 'U-02266a222036de6d8f00000000' in ardb which is not
string?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/21#issuecomment-34182984
.

Pedro Larroy Tovar | http://pedro.larroy.com/

[larroy]

Restarted server and couldn't reproduce anymore :-(

get "U-02266a222036de6d8f00000000"
"402266a222036de6d8f00000000\x00\x00\x00\x00\x00\x00\x00\x00\x04\xc1\xffV\xb2c\x13&@\xbf\x0e\x9c3\xa2|I@l&\xdfls\x13&@\xe2\xe4~\x87\xa2|I@\x00\x80@\x10$01266a222000000000$0136de6d8f00000000\xc1\xffV\xb2c\x13&@\xbf\x0e\x9c3\xa2|I@l&\xdfls\x13&@\xe2\xe4~\x87\xa2|I@"

[larroy]

I think the reply in case set fails should be "-Error message\r\n" as described in the redis protocol.

In your case you are returning $-1 when there is some error in the KV, which makes it indistinguishable from when there's an "abort" by using NX or XX

[yinqiwen]

Yes, it's a bug. I'll fix it later.

[larroy]

The reason why is $-1 seems to be also some failure in leveldb:

260│ }
261│ }
262│ else
263│ {
264│ s = m_db->Put(leveldb::WriteOptions(), LEVELDB_SLICE(key), LEVELDB_SLICE(value));
265│ }
266├> return s.ok() ? 0 : -1;
267│ }
268│ int LevelDBEngine::Get(const Slice& key, std::string* value)
269│ {
270│ leveldb::ReadOptions options;
271│ ContextHolder& holder = m_context.GetValue();
272│ options.snapshot = holder.snapshot;
273│ leveldb::Status s = m_db->Get(options, LEVELDB_SLICE(key), value);
274│ return s.ok() ? 0 : -1;
275│ }
276│ int LevelDBEngine::Del(const Slice& key)
277│ {
278│ leveldb::Status s = leveldb::Status::OK();
/map_int/home/larroy/devel/ardb/src/engine/leveldb_engine.cpp
leveldb::Status::operator= (this=0x7fffffffc8d0, s=...) at /home/larroy/devel/ardb/src/../deps/leveldb-1.15.0/include/leveldb/status.h:98
(gdb) p s
$55 = (const leveldb::Status &) @0x7fffffffc8e0: {
state_ = 0x7fff731bc4c0 "G"
}
(gdb) p state_
$56 = 0x0
(gdb) p s.state_
$57 = 0x7fff731bc4c0 "G"
(gdb) s
(gdb) p state_
$58 = 0x0
(gdb) n
(gdb) s
(gdb) s
leveldb::Status::~Status (this=0x7fffffffc8e0, _in_chrg=) at /home/larroy/devel/ardb/src/../deps/leveldb-1.15.0/include/leveldb/status.h:25
(gdb) s
ardb::LevelDBEngine::Put (this=0x7ffff64e10c0, key=..., value=...) at engine/leveldb_engine.cpp:266
(gdb) p s
$59 = {
state = 0x7ffff65f4fb0 "G"
}
(gdb) p s.ok()
$60 = false
(gdb)

I'm still investigating this.

Now the db returns error on every set:

redis 127.0.0.1:6200> SET aaaa 2
(nil)
(123.61s)

[yinqiwen]

Can you make clean & use 'make noopt' to rebuilt a server? This would compile with '-O0'.

[larroy]

I have a debug build with -O0 already. I don't know what state "#" means from leveldb.

257│ if (holder.count >= (uint32) m_cfg.batch_commit_watermark)
258│ {
259│ FlushWriteBatch(holder);
260│ }
261│ }
262│ else
263│ {
264│ s = m_db->Put(leveldb::WriteOptions(), LEVELDB_SLICE(key), LEVELDB_SLICE(value));
265│ }
266├> return s.ok() ? 0 : -1;
267│ }
268│ int LevelDBEngine::Get(const Slice& key, std::string* value)
269│ {
270│ leveldb::ReadOptions options;
271│ ContextHolder& holder = m_context.GetValue();
272│ options.snapshot = holder.snapshot;
273│ leveldb::Status s = m_db->Get(options, LEVELDB_SLICE(key), value);
274│ return s.ok() ? 0 : -1;
275│ }
276│ int LevelDBEngine::Del(const Slice& key)
277│ {
278│ leveldb::Status s = leveldb::Status::OK();
/map_int/home/larroy/devel/ardb/src/engine/leveldb_engine.cpp
(gdb) p Status
Attempt to use a type name as an expression
(gdb) print $eax
$79 = -163544936
(gdb) info frame
Stack level 0, frame at 0x7fffffffc850:
rip = 0x524036 in leveldb::DB::Put (db/db_impl.cc:1435); saved rip 0x522bb1
called by frame at 0x7fffffffc890
source language c++.
Arglist at 0x7fffffffc840, args: this=0x7ffff6507300, opt=..., key=..., value=...
Locals at 0x7fffffffc840, Previous frame's sp is 0x7fffffffc850
Saved registers:
rbx at 0x7fffffffc838, rbp at 0x7fffffffc840, rip at 0x7fffffffc848
(gdb) n
leveldb::DBImpl::Put (this=0x7ffff6507300, o=..., key=..., val=...) at db/db_impl.cc:1156
(gdb) fini
Run till exit from #0 leveldb::DBImpl::Put (this=0x7ffff6507300, o=..., key=..., val=...) at db/db_impl.cc:1156
0x000000000051c0b3 in ardb::LevelDBEngine::Put (this=0x7ffff64e10c0, key=..., value=...) at engine/leveldb_engine.cpp:264
Value returned is $80 = {state_ = 0x7fffffffc8e0 "\260X]\366\377\177"}
(gdb) n
(gdb) p s
$81 = {state_ = 0x7ffff65d5880 "#"}
(gdb) p s.state_
$82 = 0x7ffff65d5880 "#"
(gdb)

[larroy]

db) p s.state_[4]
$84 = 2 '\002'
(gdb)

enum Code {
kOk = 0,
kNotFound = 1,
kCorruption = 2,
kNotSupported = 3,
kInvalidArgument = 4,
kIOError = 5
};

Code code() const {
return (state_ == NULL) ? kOk : static_cast(state_[4]);
}

Looks like corruption, great.