Skip to content
This repository has been archived by the owner on Jul 26, 2022. It is now read-only.

Bloom filters and column families #3

Closed
jonahcohen opened this issue Jan 7, 2015 · 1 comment
Closed

Bloom filters and column families #3

jonahcohen opened this issue Jan 7, 2015 · 1 comment

Comments

@jonahcohen
Copy link

From @mdcallag:

I assume we will put many indexes in one column family as column family per index doesn't scale to a large number of indexes. When that is done we need a method for specifying the index columns that will be used in the column family. This assumes that we can use bloom filters on a prefix of the index key and that the format of an index key within RockDB is:
...

In that case the options are:

  1. no bloom filter
  2. bloom filter on index ID
  3. bloom filter on index ID, index column 1, ..., index column N-1>
  4. bloom filter on index ID, index column 1, ..., index column N>

Note that

  1. and 3) above are on a prefix of the visible index columns
    2), 3), and 4) require all indexes in the column family to have a common prefix in columns
    This requires much more discussion.
@yoshinorim yoshinorim mentioned this issue Jan 9, 2015
Closed
@yoshinorim
Copy link

https://reviews.facebook.net/D32607

@hermanlee hermanlee mentioned this issue Apr 2, 2015
Closed
@maykov maykov mentioned this issue Apr 29, 2015
Closed
@elenst elenst mentioned this issue May 24, 2015
Closed
maykov added a commit that referenced this issue Jul 23, 2015
@maykov
Testing the ASAN failure
eeb5ef2 
Summary:
I saw this intermittent ASAN failure
==662590== ERROR: AddressSanitizer: SEGV on unknown address 0x000000000098 (pc 0x00000112fc64 sp 0x7f0324c71710 bp 0x7f0324c71760 T21)
AddressSanitizer can not provide additional info.
    #0 0x112fc63 in my_pthread_fastmutex_lock /data/users/jenkins/workspace/myrocks-prod/BUILD_TYPE/ASan/CLIENT_MODE/Sync/PAGE_SIZE/32/TEST_SET/MixedOther/label/mysql/mysql/mysys/thr_mutex.c:479
    #1 0x18fb4b0 in _ZN23Dropped_indices_manager14remove_indicesERKSt13unordered_setIjSt4hashIjESt8equal_toIjESaIjEEP12Dict_manager /data/users/jenkins/workspace/myrocks-prod/BUILD_TYPE/ASan/CLIENT_MODE/Sync/PAGE_SIZE/32/TEST_SET/MixedOther/label/mysql/mysql/include/mysql/psi/mysql_thread.h:688
    #2 0x18b031a in _Z17drop_index_threadPv /data/users/jenkins/workspace/myrocks-prod/BUILD_TYPE/ASan/CLIENT_MODE/Sync/PAGE_SIZE/32/TEST_SET/MixedOther/label/mysql/mysql/storage/rocksdb/ha_rocksdb.cc:4927
    #3 0x7f0337d611e8 in _ZN6__asan10AsanThread11ThreadStartEv ??:0
    #4 0x7f03376dbfa7 in start_thread ??:0
    #5 0x7f0335a5f5bc in __clone ??:0
Thread T21 created by T0 here:
    #0 0x7f0337d5121b in pthread_create ??:0
    #1 0x18aec6a in _ZL17rocksdb_init_funcPv /data/users/jenkins/workspace/myrocks-prod/BUILD_TYPE/ASan/CLIENT_MODE/Sync/PAGE_SIZE/32/TEST_SET/MixedOther/label/mysql/mysql/storage/rocksdb/ha_rocksdb.cc:1765
    #2 0x60a039 in _Z24ha_initialize_handlertonP13st_plugin_int /data/users/jenkins/workspace/myrocks-prod/BUILD_TYPE/ASan/CLIENT_MODE/Sync/PAGE_SIZE/32/TEST_SET/MixedOther/label/mysql/mysql/sql/handler.cc:673
    #3 0xab04e0 in _ZL17plugin_initializeP13st_plugin_int /data/users/jenkins/workspace/myrocks-prod/BUILD_TYPE/ASan/CLIENT_MODE/Sync/PAGE_SIZE/32/TEST_SET/MixedOther/label/mysql/mysql/sql/sql_plugin.cc:1137
    #4 0xac8fe3 in _Z11plugin_initPiPPci /data/users/jenkins/workspace/myrocks-prod/BUILD_TYPE/ASan/CLIENT_MODE/Sync/PAGE_SIZE/32/TEST_SET/MixedOther/label/mysql/mysql/sql/sql_plugin.cc:1431
    #5 0x5ef74c in _ZL22init_server_componentsv .cc:5482
    #6 0x5f0cce in _Z11mysqld_mainiPPc .cc:6254
    #7 0x7f033597defe in __libc_start_main ??:0
    #8 0x5d86a8 in _start /home/engshare/third-party2/glibc/2.17/src/glibc-2.17/csu/../sysdeps/x86_64/start.S:123
==662590== ABORTING
It looks like the Dropped_indices_manager mutex is accessed by the thread after the mutex was cleaned up. I'm moving the cleanup of the manager to the thread in hopes that this will fix the failure.

Test Plan: ran rocksdb suite, nothing failed

Reviewers: hermanlee4

Reviewed By: hermanlee4

Differential Revision: https://reviews.facebook.net/D42909
spetrunia added a commit that referenced this issue Aug 27, 2015
@spetrunia
Fix Issue #3: SHOW TABLE STATUS shows Auto_increment=0
3a6af9f
maykov added a commit that referenced this issue Aug 27, 2015
@maykov
Testing the ASAN failure
28c20ef 
Summary:
I saw this intermittent ASAN failure
==662590== ERROR: AddressSanitizer: SEGV on unknown address 0x000000000098 (pc 0x00000112fc64 sp 0x7f0324c71710 bp 0x7f0324c71760 T21)
AddressSanitizer can not provide additional info.
    #0 0x112fc63 in my_pthread_fastmutex_lock /data/users/jenkins/workspace/myrocks-prod/BUILD_TYPE/ASan/CLIENT_MODE/Sync/PAGE_SIZE/32/TEST_SET/MixedOther/label/mysql/mysql/mysys/thr_mutex.c:479
    #1 0x18fb4b0 in _ZN23Dropped_indices_manager14remove_indicesERKSt13unordered_setIjSt4hashIjESt8equal_toIjESaIjEEP12Dict_manager /data/users/jenkins/workspace/myrocks-prod/BUILD_TYPE/ASan/CLIENT_MODE/Sync/PAGE_SIZE/32/TEST_SET/MixedOther/label/mysql/mysql/include/mysql/psi/mysql_thread.h:688
    #2 0x18b031a in _Z17drop_index_threadPv /data/users/jenkins/workspace/myrocks-prod/BUILD_TYPE/ASan/CLIENT_MODE/Sync/PAGE_SIZE/32/TEST_SET/MixedOther/label/mysql/mysql/storage/rocksdb/ha_rocksdb.cc:4927
    #3 0x7f0337d611e8 in _ZN6__asan10AsanThread11ThreadStartEv ??:0
    #4 0x7f03376dbfa7 in start_thread ??:0
    #5 0x7f0335a5f5bc in __clone ??:0
Thread T21 created by T0 here:
    #0 0x7f0337d5121b in pthread_create ??:0
    #1 0x18aec6a in _ZL17rocksdb_init_funcPv /data/users/jenkins/workspace/myrocks-prod/BUILD_TYPE/ASan/CLIENT_MODE/Sync/PAGE_SIZE/32/TEST_SET/MixedOther/label/mysql/mysql/storage/rocksdb/ha_rocksdb.cc:1765
    #2 0x60a039 in _Z24ha_initialize_handlertonP13st_plugin_int /data/users/jenkins/workspace/myrocks-prod/BUILD_TYPE/ASan/CLIENT_MODE/Sync/PAGE_SIZE/32/TEST_SET/MixedOther/label/mysql/mysql/sql/handler.cc:673
    #3 0xab04e0 in _ZL17plugin_initializeP13st_plugin_int /data/users/jenkins/workspace/myrocks-prod/BUILD_TYPE/ASan/CLIENT_MODE/Sync/PAGE_SIZE/32/TEST_SET/MixedOther/label/mysql/mysql/sql/sql_plugin.cc:1137
    #4 0xac8fe3 in _Z11plugin_initPiPPci /data/users/jenkins/workspace/myrocks-prod/BUILD_TYPE/ASan/CLIENT_MODE/Sync/PAGE_SIZE/32/TEST_SET/MixedOther/label/mysql/mysql/sql/sql_plugin.cc:1431
    #5 0x5ef74c in _ZL22init_server_componentsv .cc:5482
    #6 0x5f0cce in _Z11mysqld_mainiPPc .cc:6254
    #7 0x7f033597defe in __libc_start_main ??:0
    #8 0x5d86a8 in _start /home/engshare/third-party2/glibc/2.17/src/glibc-2.17/csu/../sysdeps/x86_64/start.S:123
==662590== ABORTING
It looks like the Dropped_indices_manager mutex is accessed by the thread after the mutex was cleaned up. I'm moving the cleanup of the manager to the thread in hopes that this will fix the failure.

Test Plan: ran rocksdb suite, nothing failed

Reviewers: hermanlee4

Reviewed By: hermanlee4

Differential Revision: https://reviews.facebook.net/D42909
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yoshinorim @jonahcohen