Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

crash_test: failures are reported when running with the new memtable #98

Closed
isaac-io opened this issue Aug 2, 2022 · 10 comments · Fixed by #99 or #188
Closed

crash_test: failures are reported when running with the new memtable #98

isaac-io opened this issue Aug 2, 2022 · 10 comments · Fixed by #99 or #188
Assignees
@hilikspdb @erez-speedb @ayulas
Labels
bug Something isn't working
Milestone
v2.2.0

Comments

@isaac-io
Copy link
Contributor

isaac-io commented Aug 2, 2022

When running the crash tests, all of them fail quickly with:

WARNING: prefix_size is non-zero but memtablerep != prefix_hash
Verification failed for column family 0 key 000000000000004500000000000000E500000000000000310000000000000070 (6402): Value not found: NotFound: 
Crash-recovery verification failed :(
No writes or ops?
Verification failed :(

as well as

***Iterator diverged from control iterator which has value 000000000001FF3A000000000000007C000000000000007C000000000000007C000000000000007C000000000000007C000000000000001F78787878787878 total_order_seek: 0 auto_prefix_mode: 0 S 000000000001E67D78 NPNPPPPPPN; total_order_seek: 0 auto_prefix_mode: 0 SFP 0000000000020034000000000000007C000000000000000A7878 ***
stderr has error message:
***iterator has value 000000000001E3E2000000000000007C000000000000007C000000000000007C000000000000007C000000000000007C000000000000007C0000000000000006***
stderr has error message:
***Control CF 1***
stderr has error message:
***Verification failed :(***
TEST FAILED. See kill option and exit code above!!!

and

WARNING: prefix_size is non-zero but memtablerep != prefix_hash
Verification failed
Latest Sequence Number: 2100250
[default] 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878 => 82840000868784858A8B88898E8F8C8D92939091969794959A9B98999E9F9C9DA2A3A0A1A6A7A4A5AAABA8A9AEAFACADB2B3B0B1B6B7B4B5BABBB8B9BEBFBCBD
[1] 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 => 7F8B01007B7A797877767574737271706F6E6D6C6B6A69686766656463626160
Internal keys in CF 'default', [00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878, 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878] (max 8)
  key 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878 seq 702931 type 1
Internal keys in CF '1', [00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878, 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878] (max 8)
  key 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 seq 2097862 type 1
  key 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878 seq 702932 type 1
[2] 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 => 7F8B01007B7A797877767574737271706F6E6D6C6B6A69686766656463626160
Internal keys in CF '2', [00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878, 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878] (max 8)
  key 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 seq 0 type 1
  key 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878 seq 0 type 1
[3] 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 => 7F8B01007B7A797877767574737271706F6E6D6C6B6A69686766656463626160
Internal keys in CF '3', [00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878, 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878] (max 8)
  key 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 seq 0 type 1
  key 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878 seq 0 type 1
[4] 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 => 7F8B01007B7A797877767574737271706F6E6D6C6B6A69686766656463626160
Internal keys in CF '4', [00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878, 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878] (max 8)
  key 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 seq 2097865 type 1
  key 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878 seq 702935 type 1
[6] 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 => 7F8B01007B7A797877767574737271706F6E6D6C6B6A69686766656463626160
Internal keys in CF '6', [00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878, 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878] (max 8)
  key 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 seq 0 type 1
  key 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878 seq 0 type 1
[8] 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 => 7F8B01007B7A797877767574737271706F6E6D6C6B6A69686766656463626160
Internal keys in CF '8', [00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878, 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878] (max 8)
  key 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 seq 2097869 type 1
  key 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878 seq 702939 type 1
Crash-recovery verification failed :(
No writes or ops?
Verification failed :(

(all of the examples are taken from here).

This is happening only when running with the new memtable, which was merged as part of #22, and enabled in crash tests with #89 + #93.

To Reproduce

make crash_test

Expected behavior

Crash test succeeds.

System

  • OS: Ubuntu 20.04
  • Compiler and version: GCC 9.4.0

Additional context

N/A
@isaac-io isaac-io self-assigned this Aug 2, 2022
isaac-io added a commit that referenced this issue Aug 2, 2022
@isaac-io
crash_test: remove the hash-based memtable from crash test (#98)
4de46bc 
The test seems to fail for now, so in order to avoid headaches for devs
working on `main`, let's not test the new memtable until the issues are
fixed. This is not intended as a fix for #98, but rather as a temporary
workaround to allow testing to continue, since no one is supposed to be
using the new memtable yet.
@isaac-io isaac-io added the bug Something isn't working label Aug 2, 2022
@isaac-io isaac-io mentioned this issue Aug 2, 2022
Merged
@isaac-io isaac-io removed their assignment Aug 2, 2022
isaac-io added a commit that referenced this issue Aug 2, 2022
@isaac-io
crash_test: remove the hash-based memtable from crash test (#98)
21c6bd5 
The test seems to fail for now, so in order to avoid headaches for devs
working on `main`, let's not test the new memtable until the issues are
fixed. This is not intended as a fix for #98, but rather as a temporary
workaround to allow testing to continue, since no one is supposed to be
using the new memtable yet.
@isaac-io isaac-io linked a pull request Aug 3, 2022 that will close this issue
crash_test: remove the hash-based memtable from crash test (#98) #99
Merged
@ayulas
Copy link
Contributor

ayulas commented Aug 15, 2022 •
edited

@hilikspdb please review 22-spdb-memtable

@ayulas
Copy link
Contributor

ayulas commented Aug 28, 2022

PR #142

@isaac-io isaac-io linked a pull request Aug 28, 2022 that will close this issue
hash spd bucket get - use key without timestamp #142
Closed
ayulas added a commit that referenced this issue Aug 28, 2022
@ayulas
Revert "crash_test: remove the hash-based memtable from crash test (#98…
4cde181 
…)"

This reverts commit 21c6bd5.
@isaac-io isaac-io assigned assaf-speedb and unassigned hilikspdb Aug 29, 2022
@assaf-speedb assaf-speedb removed their assignment Sep 5, 2022
@assaf-speedb
Copy link
Contributor

Stress tests failed:
b'WARNING: prefix_size is non-zero but memtablerep != prefix_hash\nIterator diverged from control iterator which has value 0000000000044A4700000000000000BD0000000000000050 total_order_seek: 0 auto_prefix_mode: 0 S 0000000000039E6700000000000000BD000000000000008D NPPPNPNPNNPNPPPNPNNPPNNPPPPP; total_order_seek: 0 auto_prefix_mode: 0 S 0000000000043C8700000000000000BD00000000000000DE \niterator has value 0000000000009FD700000000000000BD00000000000000E9\nControl CF default\n'
[2022-09-08 06:26:59.135318] ERROR: db_stress (pid=7384) failed before kill: exitcode=-11, signal=SIGSEGV

@isaac-io isaac-io mentioned this issue Sep 12, 2022
Closed
@isaac-io isaac-io added this to the v2.1.0 milestone Sep 21, 2022
@Yuval-Ariel
Copy link
Contributor

QA

blackbox, whitebox, asan blackbox, db fuzzer and db map fuzzer all passed on:
branch - 22-spdb-memtable-fix
hash - c97bc0e

@isaac-io
Copy link
Contributor Author

isaac-io commented Oct 2, 2022

c97bc0e is not the HEAD of #142, and there seem to be differences in hash_spd_rep.cc as well between the two commits as can be seen here. Not sure if these changes are relevant in any way, but it does mean that #142 cannot be merged since it wasn't tested.

@ayulas
Copy link
Contributor

ayulas commented Oct 2, 2022

#188

@isaac-io isaac-io linked a pull request Oct 2, 2022 that will close this issue
hash spd memtable #188
Merged
@ayulas ayulas removed a link to a pull request Oct 2, 2022
hash spd memtable #188
Merged
@ayulas ayulas linked a pull request Oct 2, 2022 that will close this issue
hash spd memtable #188
Merged
isaac-io added a commit that referenced this issue Oct 19, 2022
@isaac-io
crash_test: remove the hash-based memtable from crash test (#98)
569b61b 
The test seems to fail for now, so in order to avoid headaches for devs
working on `main`, let's not test the new memtable until the issues are
fixed. This is not intended as a fix for #98, but rather as a temporary
workaround to allow testing to continue, since no one is supposed to be
using the new memtable yet.
@Yuval-Ariel
Copy link
Contributor

QA passed on 20a75d1 22-spdb-memtable-blueberry

@Yuval-Ariel
Copy link
Contributor

QA passed on 03b830d

@isaac-io isaac-io modified the milestones: v2.1.0, v2.2.0 Oct 26, 2022
@Guyme Guyme mentioned this issue Nov 14, 2022
Closed
@Guyme
Copy link

Guyme commented Nov 14, 2022

Switch functionality #14 was added this ticket

@Guyme
Copy link

Guyme commented Nov 14, 2022

QA & Performance tests past on hash 7e5e7bc
PR #188

Image

Yuval-Ariel added a commit that referenced this issue Nov 14, 2022
@Yuval-Ariel
hash spd memtable (#98)
f2bf5c8 
Import shorten memtable switch latency (#14)
ayulas added a commit that referenced this issue Nov 14, 2022
@ayulas
hash spd memtable (#98)
3fbd6d9 
Import shorten memtable switch latency (#14)
Yuval-Ariel pushed a commit that referenced this issue Nov 14, 2022
@ayulas @Yuval-Ariel
hash spd memtable (#98)
3c534ac 
Import shorten memtable switch latency (#14)
Yuval-Ariel pushed a commit that referenced this issue Nov 15, 2022
@ayulas @Yuval-Ariel
hash spd memtable (#98)
99a3db1 
Import shorten memtable switch latency (#14)
Yuval-Ariel pushed a commit that referenced this issue Nov 23, 2022
@isaac-io @Yuval-Ariel
crash_test: remove the hash-based memtable from crash test (#98)
ebf4dcc 
The test seems to fail for now, so in order to avoid headaches for devs
working on `main`, let's not test the new memtable until the issues are
fixed. This is not intended as a fix for #98, but rather as a temporary
workaround to allow testing to continue, since no one is supposed to be
using the new memtable yet.
Yuval-Ariel pushed a commit that referenced this issue Nov 25, 2022
@isaac-io @Yuval-Ariel
crash_test: remove the hash-based memtable from crash test (#98)
35c5e62 
The test seems to fail for now, so in order to avoid headaches for devs
working on `main`, let's not test the new memtable until the issues are
fixed. This is not intended as a fix for #98, but rather as a temporary
workaround to allow testing to continue, since no one is supposed to be
using the new memtable yet.
Yuval-Ariel pushed a commit that referenced this issue Nov 25, 2022
@ayulas @Yuval-Ariel
hash spd memtable (#98)
8a85b53 
Import shorten memtable switch latency (#14)
@Yuval-Ariel Yuval-Ariel mentioned this issue Jan 19, 2023
Closed
Yuval-Ariel pushed a commit that referenced this issue Apr 30, 2023
@isaac-io @Yuval-Ariel
crash_test: remove the hash-based memtable from crash test (#98)
5e1e5ef 
The test seems to fail for now, so in order to avoid headaches for devs
working on `main`, let's not test the new memtable until the issues are
fixed. This is not intended as a fix for #98, but rather as a temporary
workaround to allow testing to continue, since no one is supposed to be
using the new memtable yet.
Yuval-Ariel pushed a commit that referenced this issue Apr 30, 2023
@ayulas @Yuval-Ariel
hash spd memtable (#98)
b69a7eb 
Import shorten memtable switch latency (#14)
Yuval-Ariel added a commit that referenced this issue May 1, 2023
@Yuval-Ariel
Rebase 8.1.1: fix for hash spd memtable (#98)
4f29ce4
Yuval-Ariel pushed a commit that referenced this issue May 4, 2023
@ayulas @Yuval-Ariel
hash spd memtable (#98)
9f0f8cc 
Import shorten memtable switch latency (#14)
Yuval-Ariel added a commit that referenced this issue May 4, 2023
@Yuval-Ariel
Rebase 8.1.1: fix for hash spd memtable (#98)
cf06aba
udi-speedb pushed a commit that referenced this issue Oct 31, 2023
@isaac-io @udi-speedb
crash_test: remove the hash-based memtable from crash test (#98)
ba1d08b 
The test seems to fail for now, so in order to avoid headaches for devs
working on `main`, let's not test the new memtable until the issues are
fixed. This is not intended as a fix for #98, but rather as a temporary
workaround to allow testing to continue, since no one is supposed to be
using the new memtable yet.
udi-speedb pushed a commit that referenced this issue Nov 13, 2023
@ayulas @udi-speedb
hash spd memtable (#98)
fbc9a38 
Import shorten memtable switch latency (#14)
udi-speedb pushed a commit that referenced this issue Nov 13, 2023
@Yuval-Ariel @udi-speedb
Rebase 8.1.1: fix for hash spd memtable (#98)
0a37b74
udi-speedb pushed a commit that referenced this issue Nov 15, 2023
@ayulas @udi-speedb
hash spd memtable (#98)
2402979 
Import shorten memtable switch latency (#14)
udi-speedb pushed a commit that referenced this issue Nov 15, 2023
@Yuval-Ariel @udi-speedb
Rebase 8.1.1: fix for hash spd memtable (#98)
4927543
udi-speedb pushed a commit that referenced this issue Dec 1, 2023
@isaac-io @udi-speedb
crash_test: remove the hash-based memtable from crash test (#98)
1d4f6d9 
The test seems to fail for now, so in order to avoid headaches for devs
working on `main`, let's not test the new memtable until the issues are
fixed. This is not intended as a fix for #98, but rather as a temporary
workaround to allow testing to continue, since no one is supposed to be
using the new memtable yet.
udi-speedb pushed a commit that referenced this issue Dec 3, 2023
@ayulas @udi-speedb
hash spd memtable (#98)
5e6ee22 
Import shorten memtable switch latency (#14)
udi-speedb pushed a commit that referenced this issue Dec 3, 2023
@Yuval-Ariel @udi-speedb
Rebase 8.1.1: fix for hash spd memtable (#98)
de835b9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hilikspdb hilikspdb

@erez-speedb erez-speedb

@ayulas ayulas

Labels
bug Something isn't working
Projects
None yet
Milestone
v2.2.0
7 participants
@isaac-io @Yuval-Ariel @hilikspdb @erez-speedb @assaf-speedb @ayulas @Guyme