Doubling space by primary key

[jonahcohen]

From @yoshinorim:

I just noticed primary keys are stored twice in RocksDB SE – as both key and value, even though
they are not needed to be stored as value.

How to repeat:

10 column primary key

create table r1 (
id1 bigint not null,
id2 bigint not null,
id3 bigint not null,
id4 bigint not null,
id5 bigint not null,
id6 bigint not null,
id7 bigint not null,
id8 bigint not null,
id9 bigint not null,
id10 bigint not null,
id11 bigint not null,
id12 bigint not null,
primary key (id1, id2, id3, id4, id5, id6, id7, id8, id9, id10)
) engine=rocksdb charset=latin1 collate=latin1_bin;

1 column primary key

create table r2 (
id1 bigint not null,
id2 bigint not null,
id3 bigint not null,
id4 bigint not null,
id5 bigint not null,
id6 bigint not null,
id7 bigint not null,
id8 bigint not null,
id9 bigint not null,
id10 bigint not null,
id11 bigint not null,
id12 bigint not null,
primary key (id1)
) engine=rocksdb charset=latin1 collate=latin1_bin;

Inserting 2 million rows on these tables. Values sequentially increasing from 1 to 2 million (all columns have same values per row) -- can be generated like this.
for(my $i=1; $i<= 2000000; $i++) {
for(my $j=0; $j < 12; $j++) {
if($j < 11) {
print "$i,";
}else {
print "$i\n";
}
}
}

then make compaction happen.

Size in KB:
id1-id10 pk:
353764

id1 pk:
212984

innodb
id1-id10 pk:
264241

id1 pk:
260046

So basically long primary keys increase on-disk space.

I also checked via gdb (setting breakpoint at rocksdb::WriteBatch::Put()) and confirmed.
insert into r1 values (1,1,1,1,1,1,1,1,1,1,1,1);
=>
(gdb) p key.size_
$1 = 84
(gdb) p value.size_
$2 = 96

insert into r2 values (1,1,1,1,1,1,1,1,1,1,1,1);
=>
(gdb) p key.size_
$3 = 12
(gdb) p value.size_
$4 = 96

I expected value.size_ were 12 and 84 respectively. Since both key fields and value fields are stored, duplicate fields can be removed from value fields.

[spetrunia]

I have made a fix for this. I pushed it into this branch:
https://github.com/MySQLOnRocksDB/mysql-5.6/tree/webscalesql-5.6.21.55-issue14

The branch has three revisions:

1. Fix the testsuite (It was broken by somebody who migrated to MySQLOnRocksDB repo. This is not related to this bug): 703e9a6

2. A patch that removes a piece of RocksDB-SE code that is no longer used: a4e83ad

3. The fix itself: 037e3e9 . There are no specific tests in the patch, the test coverage is that tests in t/rocksdb*.test and suite/rocksdb tests pass after the change.

Do I just push this to the main branch, or I need to submit patches to review into reviews.facebook.net ? (I haven't yet figured out how to do that)

[yoshinorim]

Thanks Sergey, please submit patches to review.facebook.net (follow instructions at https://github.com/MySQLOnRocksDB/mysql-5.6/wiki).