Create information_schema.index_file_mappings to show index id and sst filename mappings

[yoshinorim]

It is often useful to show mappings between table/index name and sst files. We already have an I_S to show mappings between index id and db.table.index name, but we don't have I_S to show mappings between index_id and sst filename yet.
There is N:N relationship between index_id and sst filename. They're inserted when new sst file is created, and deleted/inserted on compactions. sst files are immutable so mappings are not updated in place.
I think we need two steps to make this feature available.

1 Create data dictionary to keep track of index id and sst filename

dict 1: key: (sst filename, index id), value: N/A
dict 2: key: (index id, sst filename), value N/A

We may search by either sst filename or index id, so having two indexes may make sense.

For adding mappings, MyRocksTablePropertiesCollector would be the best place to do that.

For deleting mappings, MyRocks somehow needs to know which sst file was removed, then getting all associated index ids from dict1, then remove from dict2.

2 Create information_schema.index_file_mappings to show mappings between index id and sst filename

Once we have data dictionary, this is easy.

[siying]

Can https://reviews.facebook.net/D38865 and https://reviews.facebook.net/D38865 be used to implement it?

[maykov]

It would also be extremely useful to expose table properties for each sst file in one of these mappings.

[jkedgar]

I'll take a look at this.

[maykov]

There is a few confusions here:
Data dictionaries: I assumed, Yoshi used this to mean std::map, or any other in-memory data structure which implements associative array. There is no need for it to be stored.

As Jay mentioned on the group, there is no need to maintain this information since it can be easily requested from RocksDB. You can use db::GetPropertiesOfAllTables to get a list of all sst files with their properties. Properties will contain index id's. If we need table names, we can start storing them in table properties as well.

[jkedgar]

It turns we have some per SST stats already available. I was able to leverage that information (thanks Alexey!) and I have this working. Alexey had asked for more table properties, so I have row count and data size along with the index id and sst name. Here's how it currently looks after running a simple test where I inserted 6 rows into a trivial table (index id 256) as well as running sysbench for a few minutes (index ids 258 and 259) - index id's 1, 2, 3, 5, and 7 have to do with system indexes.

mysql> select * from information_schema.rocksdb_index_file_map;
+------------+----------+----------+-----------+
| SST_NAME   | INDEX_ID | NUM_ROWS | DATA_SIZE |
+------------+----------+----------+-----------+
| 000052.sst |      258 |    10000 |   1920000 |
| 000052.sst |      259 |    14268 |    171216 |
| 000009.sst |      256 |        3 |        24 |
| 000039.sst |      256 |        3 |        24 |
| 000012.sst |        1 |        1 |        18 |
| 000012.sst |        2 |        1 |        14 |
| 000012.sst |        3 |        1 |        14 |
| 000012.sst |        7 |        1 |        10 |
| 000055.sst |        1 |        3 |        76 |
| 000055.sst |        2 |        3 |        36 |
| 000055.sst |        5 |        1 |         8 |
| 000055.sst |        7 |        1 |        10 |
+------------+----------+----------+-----------+
12 rows in set (0.00 sec)

mysql> select * from information_schema.rocksdb_index_file_map order by index_id, sst_name;
+------------+----------+----------+-----------+
| SST_NAME   | INDEX_ID | NUM_ROWS | DATA_SIZE |
+------------+----------+----------+-----------+
| 000012.sst |        1 |        1 |        18 |
| 000055.sst |        1 |        3 |        76 |
| 000012.sst |        2 |        1 |        14 |
| 000055.sst |        2 |        3 |        36 |
| 000012.sst |        3 |        1 |        14 |
| 000055.sst |        5 |        1 |         8 |
| 000012.sst |        7 |        1 |        10 |
| 000055.sst |        7 |        1 |        10 |
| 000009.sst |      256 |        3 |        24 |
| 000039.sst |      256 |        3 |        24 |
| 000052.sst |      258 |    10000 |   1920000 |
| 000052.sst |      259 |    14268 |    171216 |
+------------+----------+----------+-----------+
12 rows in set (0.00 sec)

mysql> select index_id, count(sst_name) from information_schema.rocksdb_index_file_map group by index_id;
+----------+-----------------+
| index_id | count(sst_name) |
+----------+-----------------+
|        1 |               2 |
|        2 |               2 |
|        3 |               1 |
|        5 |               1 |
|        7 |               2 |
|      256 |               2 |
|      258 |               1 |
|      259 |               1 |
+----------+-----------------+
8 rows in set (0.00 sec)

Any comments?

[jkedgar]

Closed with commit 5f4b759