Index Condition Pushdown for RocksDB-SE

[spetrunia]

We need to support Index Condition Pushdown in RocksDB-SE.

[spetrunia]

Got a patch that seems to work. Testing is difficult, because there is no counter for how many index records were scanned.

MariaDB has counters that tell how many times Index Condition was checked ( https://mariadb.com/kb/en/mariadb/index-condition-pushdown/), but MySQL 5.6 (and so webscale) doesn't.

[spetrunia]

https://reviews.facebook.net/D35181

[spetrunia]

It seems, internal_key_skipped count doesn't count all rows that were read from an index. The following query makes 10 Next() calls, but nothing increases:

MySQL [test]> SELECT value FROM INFORMATION_SCHEMA.ROCKSDB_PERF_CONTEXT WHERE TABLE_NAME='t2' and table_schema='test' and STAT_TYPE='INTERNAL_KEY_SKIPPED_COUNT';
+-------+
| value |
+-------+
|    20 |
+-------+
1 row in set (0.00 sec)

MySQL [test]> set optimizer_switch='index_condition_pushdown=off';
Query OK, 0 rows affected (0.00 sec)

MySQL [test]> select * from t2 where kp1 between 1 and 10 ;
+----+------+------+------+
| pk | kp1  | kp2  | col1 |
+----+------+------+------+
|  1 |    1 |    1 |    1 |
|  2 |    2 |    2 |    2 |
|  3 |    3 |    3 |    3 |
|  4 |    4 |    4 |    4 |
|  5 |    5 |    5 |    5 |
|  6 |    6 |    6 |    6 |
|  7 |    7 |    7 |    7 |
|  8 |    8 |    8 |    8 |
|  9 |    9 |    9 |    9 |
| 10 |   10 |   10 |   10 |
+----+------+------+------+
10 rows in set (7.36 sec)

MySQL [test]> SELECT value FROM INFORMATION_SCHEMA.ROCKSDB_PERF_CONTEXT WHERE TABLE_NAME='t2' and table_schema='test' and STAT_TYPE='INTERNAL_KEY_SKIPPED_COUNT';
+-------+
| value |
+-------+
|    20 |
+-------+
1 row in set (0.00 sec)

[hermanlee]

Actually, stepping through gdb, it looks like the counter is getting incremented, but we're not recording it. I've set a breakpoint where the counter is bumped and it's the following stack trace:

(gdb) bt
#0 rocksdb::DBIter::FindNextUserEntryInternal (this=this@entry=0x7fffe3489a38, skipping=skipping@entry=true) at db/db_iter.cc:225
#1 0x0000000000c32a98 in FindNextUserEntry (skipping=true, this=0x7fffe3489a38) at db/db_iter.cc:203
#2 rocksdb::DBIter::Next (this=0x7fffe3489a38) at db/db_iter.cc:190
#3 0x0000000000bfc59f in Apply_changes_iter::Next (this=0x7fffe343b0a0) at /home/herman/rocksdb-mysql/5.6/storage/rocksdb/rdb_applyiter.cc:62
#4 0x0000000000bdbb86 in ha_rocksdb::index_next_with_direction (this=0x7fffe3457510, buf=0x7fffe3448010 "\371\001", move_forward=true)
at /home/herman/rocksdb-mysql/5.6/storage/rocksdb/ha_rocksdb.cc:3280
#5 0x0000000000bdba86 in ha_rocksdb::index_next (this=0x7fffe3457510, buf=0x7fffe3448010 "\371\001")
at /home/herman/rocksdb-mysql/5.6/storage/rocksdb/ha_rocksdb.cc:3252

We're only recording stats in the secondary_index_read() and rnd_next_with_direction() spots, but not in the apply iterator, which calls into rocksdb directly. I can move perf records up a level and have them at the index_first/index_read_map and rnd_init/rnd_first calls. How does that sound?