Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Gemini failing after warmup period #5509

Closed
KnifeyMoloko opened this issue Nov 30, 2022 · 4 comments
Closed

Gemini failing after warmup period #5509

KnifeyMoloko opened this issue Nov 30, 2022 · 4 comments
Assignees
@KnifeyMoloko

Comments

@KnifeyMoloko
Copy link
Contributor

KnifeyMoloko commented Nov 30, 2022 •
edited
Loading

Installation details

Kernel Version: 5.15.0-1023-aws
Scylla version (or git commit hash): 5.0.6-20221122.d83134a24 with build-id 81ebbffa251c189e695922f683c60725c3df23cb
Reprducer: https://jenkins.scylladb.com/view/scylla-5.0/job/scylla-5.0/job/SCT_Reproducers/job/gemini-reproduce-gemini-3h-with-nemesis-test-36/

Cluster size: 3 nodes (i3.large)

Scylla Nodes used in this run:

  • gemini-with-nemesis-3h-normal-5-0-oracle-db-node-c116e017-1 (100.26.149.39 | 10.0.2.238) (shards: 2)
  • gemini-with-nemesis-3h-normal-5-0-db-node-c116e017-4 (18.208.196.33 | 10.0.2.94) (shards: 2)
  • gemini-with-nemesis-3h-normal-5-0-db-node-c116e017-3 (18.206.136.207 | 10.0.3.23) (shards: 2)
  • gemini-with-nemesis-3h-normal-5-0-db-node-c116e017-2 (3.238.145.35 | 10.0.3.181) (shards: 2)
  • gemini-with-nemesis-3h-normal-5-0-db-node-c116e017-1 (3.236.131.45 | 10.0.1.150) (shards: 2)

OS / Image: ami-067e7db87ece5975f (aws: us-east-1)

Test: gemini-3h-with-nemesis-test
Test id: c116e017-a6b7-49c4-a952-d7557c533c69
Test name: scylla-5.0/gemini-/gemini-3h-with-nemesis-test
Test config file(s):

Issue description

>>>>>>>
Gemini warmup phase failed with data difference between oracle and test db. I've broken down the results below so they are a bit more readable.

2022-11-28 12:59:10.697: (TestFrameworkEvent Severity.ERROR) period_type=one-time event_id=d4c95b01-c1d0-4938-8a59-a095ad8dbb6f, source=GeminiTest.test_load_random_with_nemesis (gemini_test.GeminiTest)() message=Traceback (most recent call last):
File "/home/ubuntu/scylla-cluster-tests/gemini_test.py", line 68, in test_load_random_with_nemesis
self.verify_results()
File "/home/ubuntu/scylla-cluster-tests/gemini_test.py", line 127, in verify_results
self.fail(self.gemini_results['results'])

+map:

('[ck0:-27190 ck1:0.55236286 col0:map[udt_3625021091_0:91 '
 'udt_3625021091_1:28644100-0da3-11e9-9b6a-02c6da6b609d '
 'udt_3625021091_2:140.114.244.139] '
 'col10:TDH4ewLVOfIKoRW9lXND0QBp346G2gkM1QyCx56bjA5DjBboeXtjomyai6TcVsSW5nPdxYgRuh1m9gGoupnBgs88YenZzuB7U+wcA/7iHLGyGpGJPyuRdcZwjM3JWzQR1Qfz0mzbU79F8NSDYppTDglI9bcVIHJ2o2ZLuISsaCIulRjCjE874fFnwdhrpHd5V3DBYpFA/JlPoCRNfwgNZaRZZzDwkKubvgX008rRTWafLCKE6+XzLF/UrM2WoVdZvk2CSVNCgbjjwGc3VNZKAWjyODw9+wlr0hwaVKuI2zNTLJxqBcgzu1hkYfMwpV0hg6lOICmWOMZsIBtLnFvZXYIDA5rrdVKLDfIFW/7Og1+qed2DaDOo4gJDe6SHW+g4vB5z3G3fc4370QB3/ZRpKvtuXsc5AhxKMqKNZduhzgEfNUw1loEZX0RRi+zU0Ch132X5mILC1E3AeGfe3NU0gjhqJ5wUMEG0e56Koh+ZEkT7rtuGxp9EQk09wRT6GHRTpDei/RHLkj6Quv4gmDu5dP4qC/8Wxb560+r2EPgtEWYxSQpJ1ije/3JzbftEV7N8WprxsqaugXJudMgAnUznh5r0q0UuxfspGcFIbWHsDDluMurQhS9d28vQlo5iXn8fBTjgJTTYd3BIrAeUuEu9cC2IzZkB1z1yiGcxeEJvqlfNj5jbO6uFpCiBtRGtvwuv83IvpT885Ko9WGSTHMwiY6I3cUDMao1M/kefVKL/wMTpeUtsVZl '
 'col11:0.76346874 col12:0.4599679 '
 'col1[0]:7dbb7c80-8991-11d5-9b6c-02c6da6b609d '
 'col1[10]:PKtc1rPi0NFuWaICoXJ1d3K7XYIwhg4rrwUzV3SGG+U7O7mzPRHV937WN9jeWjZ9amaMUS7KpQAIFcWJKhHyz7xsy6UI3mhuww65zXf7RYAihqZJw2TvGFXHqJip6AmvDdXBIiLSrDmve1w71dsJ27NZHiDokF0uD882WGRjOkLC0RQs8IDRYHERHioOWXix9BFCueFyJX5YV0AFCKE8fF9n6/J8hx2aOqqfiIYoEMtNQfmxXKbvedDr4BzM6N48ZDWiXyJnuKyhQ0tnI3tPlpC06ZaZXbnsNddyvHDEPRVyNv0ADNl6mInylZ4b63hFWKgHyijIKYxJ5VdcozjZwKhcHhB9auKWRokSFZ0NX2rWJTYSVa8GsHUfrUHucx5Mfi6cc9W5RVXh7nPdQIlv+E3u98B+9+GHrljKaUCH9+H2ErsrEvGxpVB+k8rnSZgR6XDZelnIF6T6RWDprykZznM8nXGDc01r9oUWj4m1s0TiWVWF3XrW/rZFDFrU6FibLNdNYGuiJAiqOnSaDzFaOThgeShFeUmkYJEzRkPL/3EVGOyLhW3lScoyGjRHEd/ccg+qfaBErtxNJAcCVoku+HOCCMtIWpbJxjJikFIENRgqP6G7saJ0SUtL8QmLyPOej7d9On/GFfbbdzyIYi/GFqtfsArRW75w8Ced/l1n '
 'col1[1]:2000-03-18 00:00:00 +0000 UTC '
 'col1[2]:e4672680-00ef-11da-9b6e-02c6da6b609d '
 'col1[3]:P4lv63N1yc1KvmIiXHgvmcfDWvopLCXFNTK2rfX4Cy2vqVWh7lNMvGbiasRDIke7oOPHVzPxBLnW1CrxByhJWLWPYybzo2k0SKK82ZntEm/yMadaUDvYUagRtmd+NN1O3OQP2RBa98EVkrnwKsLIsmFFaw87pPBKFD00Ii5ImKusIT1DfIuZ5KB20nJpe+dT2LUaFsLL9ojaS1zfBUuwf2jDPP7nC/ctaxz2G+XTyCyUE+Rmb9tlb6JzQlpBVYMuGzuujMcUogclDhWDZDJ5orGjBSCfSIKLInDOtGNFjfGVDZKSZ4VOoSpo3TcYs65xh3lNZ+lsGBbsP8PnQdVMDnv7QexUzZWm6gnJOM1V4teqcXUPg1BtYzNTnzqt5xHUvV3NLCFux+EazEACtQBAR+vklG+IFCMXZYYV '
 'col1[4]:16.20.206.225 '
 'col1[5]:27oXmu1KrvqjUOd6ZwFDW8hmCKm3CEMWXgj+v2bRaxwGgZ5ICim0v0DBwN5U/MhizHBv95ttUu16S2X2a7vZwKMN73FTCvfIsJI0QyfgMg3xtrUCn0sk4xSEYeGFgdAKCfoVPj+/H1JCNJpPS/hWpR1bkPR773oK6WPKF8yBhMGhrlnIFAi7DyBHibiYv8CjkAw/YYtQS3ayb5tHejhZ50Uy0vevuOga+2T0t1rkuYnzLotRTjgn6M6jtvfnSzuT3gykBZ+IFZbANDIFvlkQwp+SWT1PVBQ1thDPhGMEMRv8CHJ8Dq0sXtKRh1qk4003095wP5f+e3MmIiiylTSGlrVryvjmsn1PdjOBdU52MqHa1O/dQweB8U+0tY5jAaTkoqrmF0hiIYcwUi9q+7pyussL95IghnJfEJdmhLQv8CQBjQOa5D7oPLsehYUK8SyEkI6YqFRGB/NAAUKQlSow75/zNzNXxeC6hvOn/vI1PpSTI5TvSsbxptx/6rRRuoXV/B5ZU8T+gSUaE1LJaz0HxhYjE+Ejdi2RCdMqkd0aju9rmz3BVXQlaH0/9wBZizNxxMbUKeIFKw4dgvzUQdfiDPqQ/cVOkn67/oJfpdWsbii+wbXEq7ysoS7TWJoaRt9vnA0CX9f5yJAhs8Fl2ICHkVyFTSoeO1K1enljLG5cXFWTrfPPAHD+4tBnYSHWYGnyS5VMiA/xOnY8B82y0uEqGPZvkRu1YAULQG57STpu6mUBgDCZja+oWX9Zr9fc8m2w/T '
 'col1[6]:1979-03-06 00:00:00 +0000 UTC '
 'col1[7]:1jU2JJcq0I1gGnOZm9ZtwUP9CIq5HEh+oG6alwM+0rSc5c6IQBvtXs8oAdo/8ZG1VXtvnTMbtHFiQRVyz3omv2cCmZE960ZzFK3mF34SbL+I7/VmhqhrSD1aKR7il2z6yUVHgkJTquiUVMNygNOqfkWvpFKToK8PJN9fUnNRlolmKiGiCYupFG4gb1jCuyiRxAYDqvefRZLvdyyu32VfPSCRFFydVo6UlDZch6qDsdA2zjV9PLotnGpYzKA9MzjShiNeVIIGrXYRoxJbDdsLrP3R12M16I67Atslc2YaWcJMfm3fFSZ94ZqjroI7H2ltkNo0un3PUAzdQpYCuZAz1j9UgCZevIUEmBOJdn45V5EJ8aXJgfxXWSDVIicuNT/i2YCqm0lPgDTqWhHHcSa86C9sjIeD2uT+ '
 'col1[8]:218.169.229.103 col1[9]:81.102.49.208 '
 'col2:U0NUdWVIyeYrAb3BlzvRMn6dUux9rii0UFEiH+LHzYP col3:927484090770182872 '
 'col4:map[5986897694310434673:RODStjHDVvOtzVGXtTOzcuYU33q5pVZULxoMmVPR7oPPFhEhKGGnh7t0DmmdoEE5/AUBwCb2rcF0aQXLDjnCZSj8pXzqanhCrZK4RSO+zcDRb7BfpeOU7JrEKJHjuj5YVmlmfMb4p5DX5s4qaWpNwL5Go51KUDGETzTXQ+Y3+vJ5m86UsycLd0vPSZJIUnTobMDs6glLaZ8m5/tFjtpc5RjAXtNgE08+HwINMv1V4OAUEe6ZSGmZnx2LqGNBnnMQUZjRRnZDvHJpxG3LOPYdaNYyjSOfxbqrGcAyU+Nag0Fl9sOEkDLwZL4D7/U6XKBbIaut86t5U+SoDz/sTne9ZXWD6F32A1AzUJYTvcmTaWf9scLVm908HB2MrYL2GA2iPIozNWqwWQeQtFEIkb3H172Tbqafva2yAp7VuswQVWGsIqXykagwAHMDw7voDYgZA6qvGWWoUnFzdsu8dCg39o8xw5ubVABNn0vflbkxzoF2R7CycJHfA] '
 'col5:4006430191304441957 col6:5313336783095743104 col7:2011-08-26 00:00:00 '
 '+0000 UTC col8:map[udt_3663736436_0:925b4980-d71b-11d1-9b73-02c6da6b609d '
 'udt_3663736436_1:[53 53 54 97 53 50 55 54 53 97 54 50 54 53 53 49 54 98 55 '
 '56 54 98 54 101 54 53 52 100 53 51 55 49 55 55 53 54 51 49 54 99 54 51 55 55 '
 '51 54 54 98 52 100 55 54 51 50 52 100 52 49 51 56 52 56 52 99 53 97 55 48 55 '
 '97 51 55 54 99 54 101 55 56 54 51 54 101 51 56 54 97 53 50 53 48 54 97 55 48 '
 '55 57 55 57 50 98 50 102 51 54 55 51 53 49 54 50 54 102 55 54 54 49 52 100 '
 '55 52 52 102 54 54 55 49 53 48 50 98 55 54 50 102 54 53 54 57 54 55 55 54 51 '
 '48 50 102 52 101 52 49 54 101 50 98 51 49 54 52 50 98 52 53 55 56 53 51 51 '
 '49 51 53 53 49 54 102 55 51 52 57 55 97 52 101 54 100 52 52 52 54 55 50 52 '
 '50 51 49 52 50 53 55 54 101 51 50 55 50 54 101 54 102 55 51 53 54 54 50 54 '
 '102 54 99 54 54 54 53 53 48 54 52 51 54 52 49 55 49 55 50 54 99 53 50 51 50 '
 '55 52 50 102 54 53 55 49 54 57 52 51 54 57 54 50 51 48 51 52 52 52 52 50 50 '
 '102 54 49 54 53 52 52 51 49 51 50 53 97 51 50 54 102 53 56 52 49 52 101 53 '
 '54 53 57 54 57 51 48 52 51 51 52 54 57 51 48 54 101 52 99 51 53 53 50 52 101 '
 '52 49 55 50 53 48 51 57 55 52 52 49 52 56 55 51 54 101 52 53 54 99 53 51 55 '
 '51 50 98 50 102 54 53 51 56 52 56 53 49 52 51 53 51 52 97 52 51 54 97 55 49 '
 '54 53 52 54 53 57 53 51 52 49 53 50 53 50 53 97 54 50 51 48 52 100 53 57 53 '
 '54 54 50 53 55 55 48 53 49 52 52 54 50 54 55 54 97 52 101 55 57 50 98 50 102 '
 '54 54 54 100 52 99 52 102 54 54 54 54 54 98 51 48 53 52 52 98] '
 'udt_3663736436_2:-69 udt_3663736436_3:false '
 'udt_3663736436_4:487d9400-d593-11cc-9b71-02c6da6b609d '
 'udt_3663736436_5:0.23399164 '
 'udt_3663736436_6:a5ef9580-6ab8-11bc-9b72-02c6da6b609d udt_3663736436_7:{0 0 '
 '2280000000000} '
 'udt_3663736436_8:RDu3MQ1H7B8VuWx3pkXYhxRC1n5KCobS/f1znvhTqF+W4C7+uPGTNQzIQHORCJqhQ5KjeWJv8LgD6JlxbKmnQQ/rS1sQWp9juCaQU+qIkj3od9NjeSgW2c6FPSkz9h7xyDi0ITVko2fwG9hHS9XSM1pJvAn+AZYVs7i6qXz1m9nBwubyztQrkimoUYsDGkJxngu4ucV2YQ5bTqiJrvyNvfIv7JpOAnZv6zZ8VNZec7DZPyUjP1mhIHDDiGSeTlxaRHBbyZh+muQztGyS6zTVOI] '
 'col9:0.2268903 pk0:18980 pk1:26005 pk2:-71]):   map[string]interface{}{\n'
 '  \t... // 17 identical entries\n'
 '  \t"col2": string("U0NUdWVIyeYrAb3BlzvRMn6dUux9rii0UFEiH+LHzYP"),\n'
 '  \t"col3": s"927484090770182872",\n'
 '  \t"col4": map[*big.Int]string{\n'
 '- \t\ts"5986897694310434673": '
 '"RODStjHDVvOtzVGXtTOzcuYU33q5pVZULxoMmVPR7oPPFhEhKGGnh7t0DmmdoEE5/AUBwCb2rcF0aQXLDjnCZSj8pXzqanhCrZK4RSO+zcDRb7BfpeOU7JrEKJHjuj5YVmlmfMb4p5DX5s4qaWpNwL5Go51KUDGETzTXQ+Y3+vJ5m86UsycLd0vPSZJIUnTobMDs6glLaZ8m5/tFjtpc5RjAXtNgE08+HwINMv1V4OAUEe6ZSGmZnx2LqGNBnnMQUZjRRnZDvHJpxG3LOPYdaNYyjSOfxbqrGcAyU+Nag0Fl9sOEkDLwZL4D7/U6XKBbIaut86t5U+SoDz/sTne9ZXWD6F32A1AzUJYTvcmTaWf9scLVm908HB2MrYL2GA2iPIozNWqwWQeQtFEIkb3H172Tbqafva2yAp7VuswQVWGsIqXykagwAHMDw7voDYgZA6qvGWWoUnFzdsu8dCg39o8xw5ubVABNn0vflbkxzoF2R7CycJHfA",\n'
 '+ \t\ts"5986897694310434673": '
 '"RODStjHDVvOtzVGXtTOzcuYU33q5pVZULxoMmVPR7oPPFhEhKGGnh7t0DmmdoEE5/AUBwCb2rcF0aQXLDjnCZSj8pXzqanhCrZK4RSO+zcDRb7BfpeOU7JrEKJHjuj5YVmlmfMb4p5DX5s4qaWpNwL5Go51KUDGETzTXQ+Y3+vJ5m86UsycLd0vPSZJIUnTobMDs6glLaZ8m5/tFjtpc5RjAXtNgE08+HwINMv1V4OAUEe6ZSGmZnx2LqGNBnnMQUZjRRnZDvHJpxG3LOPYdaNYyjSOfxbqrGcAyU+Nag0Fl9sOEkDLwZL4D7/U6XKBbIaut86t5U+SoDz/sTne9ZXWD6F32A1AzUJYTvcmTaWf9scLVm908HB2MrYL2GA2iPIozNWqwWQeQtFEIkb3H172Tbqafva2yAp7VuswQVWGsIqXykagwAHMDw7voDYgZA6qvGWWoUnFzdsu8dCg39o8xw5ubVABNn0vflbkxzoF2R7CycJHfA",\n'
 '  \t},\n'
 '  \t"col5": s"4006430191304441957",\n'
 '  \t"col6": int64(5313336783095743104)')

-map:

('[ck0:-27190 ck1:0.55236286 col0:map[udt_3625021091_0:91 '
 'udt_3625021091_1:28644100-0da3-11e9-9b6a-02c6da6b609d '
 'udt_3625021091_2:140.114.244.139] '
 'col10:TDH4ewLVOfIKoRW9lXND0QBp346G2gkM1QyCx56bjA5DjBboeXtjomyai6TcVsSW5nPdxYgRuh1m9gGoupnBgs88YenZzuB7U+wcA/7iHLGyGpGJPyuRdcZwjM3JWzQR1Qfz0mzbU79F8NSDYppTDglI9bcVIHJ2o2ZLuISsaCIulRjCjE874fFnwdhrpHd5V3DBYpFA/JlPoCRNfwgNZaRZZzDwkKubvgX008rRTWafLCKE6+XzLF/UrM2WoVdZvk2CSVNCgbjjwGc3VNZKAWjyODw9+wlr0hwaVKuI2zNTLJxqBcgzu1hkYfMwpV0hg6lOICmWOMZsIBtLnFvZXYIDA5rrdVKLDfIFW/7Og1+qed2DaDOo4gJDe6SHW+g4vB5z3G3fc4370QB3/ZRpKvtuXsc5AhxKMqKNZduhzgEfNUw1loEZX0RRi+zU0Ch132X5mILC1E3AeGfe3NU0gjhqJ5wUMEG0e56Koh+ZEkT7rtuGxp9EQk09wRT6GHRTpDei/RHLkj6Quv4gmDu5dP4qC/8Wxb560+r2EPgtEWYxSQpJ1ije/3JzbftEV7N8WprxsqaugXJudMgAnUznh5r0q0UuxfspGcFIbWHsDDluMurQhS9d28vQlo5iXn8fBTjgJTTYd3BIrAeUuEu9cC2IzZkB1z1yiGcxeEJvqlfNj5jbO6uFpCiBtRGtvwuv83IvpT885Ko9WGSTHMwiY6I3cUDMao1M/kefVKL/wMTpeUtsVZl '
 'col11:0.76346874 col12:0.4599679 '
 'col1[0]:7dbb7c80-8991-11d5-9b6c-02c6da6b609d '
 'col1[10]:PKtc1rPi0NFuWaICoXJ1d3K7XYIwhg4rrwUzV3SGG+U7O7mzPRHV937WN9jeWjZ9amaMUS7KpQAIFcWJKhHyz7xsy6UI3mhuww65zXf7RYAihqZJw2TvGFXHqJip6AmvDdXBIiLSrDmve1w71dsJ27NZHiDokF0uD882WGRjOkLC0RQs8IDRYHERHioOWXix9BFCueFyJX5YV0AFCKE8fF9n6/J8hx2aOqqfiIYoEMtNQfmxXKbvedDr4BzM6N48ZDWiXyJnuKyhQ0tnI3tPlpC06ZaZXbnsNddyvHDEPRVyNv0ADNl6mInylZ4b63hFWKgHyijIKYxJ5VdcozjZwKhcHhB9auKWRokSFZ0NX2rWJTYSVa8GsHUfrUHucx5Mfi6cc9W5RVXh7nPdQIlv+E3u98B+9+GHrljKaUCH9+H2ErsrEvGxpVB+k8rnSZgR6XDZelnIF6T6RWDprykZznM8nXGDc01r9oUWj4m1s0TiWVWF3XrW/rZFDFrU6FibLNdNYGuiJAiqOnSaDzFaOThgeShFeUmkYJEzRkPL/3EVGOyLhW3lScoyGjRHEd/ccg+qfaBErtxNJAcCVoku+HOCCMtIWpbJxjJikFIENRgqP6G7saJ0SUtL8QmLyPOej7d9On/GFfbbdzyIYi/GFqtfsArRW75w8Ced/l1n '
 'col1[1]:2000-03-18 00:00:00 +0000 UTC '
 'col1[2]:e4672680-00ef-11da-9b6e-02c6da6b609d '
 'col1[3]:P4lv63N1yc1KvmIiXHgvmcfDWvopLCXFNTK2rfX4Cy2vqVWh7lNMvGbiasRDIke7oOPHVzPxBLnW1CrxByhJWLWPYybzo2k0SKK82ZntEm/yMadaUDvYUagRtmd+NN1O3OQP2RBa98EVkrnwKsLIsmFFaw87pPBKFD00Ii5ImKusIT1DfIuZ5KB20nJpe+dT2LUaFsLL9ojaS1zfBUuwf2jDPP7nC/ctaxz2G+XTyCyUE+Rmb9tlb6JzQlpBVYMuGzuujMcUogclDhWDZDJ5orGjBSCfSIKLInDOtGNFjfGVDZKSZ4VOoSpo3TcYs65xh3lNZ+lsGBbsP8PnQdVMDnv7QexUzZWm6gnJOM1V4teqcXUPg1BtYzNTnzqt5xHUvV3NLCFux+EazEACtQBAR+vklG+IFCMXZYYV '
 'col1[4]:16.20.206.225 '
 'col1[5]:27oXmu1KrvqjUOd6ZwFDW8hmCKm3CEMWXgj+v2bRaxwGgZ5ICim0v0DBwN5U/MhizHBv95ttUu16S2X2a7vZwKMN73FTCvfIsJI0QyfgMg3xtrUCn0sk4xSEYeGFgdAKCfoVPj+/H1JCNJpPS/hWpR1bkPR773oK6WPKF8yBhMGhrlnIFAi7DyBHibiYv8CjkAw/YYtQS3ayb5tHejhZ50Uy0vevuOga+2T0t1rkuYnzLotRTjgn6M6jtvfnSzuT3gykBZ+IFZbANDIFvlkQwp+SWT1PVBQ1thDPhGMEMRv8CHJ8Dq0sXtKRh1qk4003095wP5f+e3MmIiiylTSGlrVryvjmsn1PdjOBdU52MqHa1O/dQweB8U+0tY5jAaTkoqrmF0hiIYcwUi9q+7pyussL95IghnJfEJdmhLQv8CQBjQOa5D7oPLsehYUK8SyEkI6YqFRGB/NAAUKQlSow75/zNzNXxeC6hvOn/vI1PpSTI5TvSsbxptx/6rRRuoXV/B5ZU8T+gSUaE1LJaz0HxhYjE+Ejdi2RCdMqkd0aju9rmz3BVXQlaH0/9wBZizNxxMbUKeIFKw4dgvzUQdfiDPqQ/cVOkn67/oJfpdWsbii+wbXEq7ysoS7TWJoaRt9vnA0CX9f5yJAhs8Fl2ICHkVyFTSoeO1K1enljLG5cXFWTrfPPAHD+4tBnYSHWYGnyS5VMiA/xOnY8B82y0uEqGPZvkRu1YAULQG57STpu6mUBgDCZja+oWX9Zr9fc8m2w/T '
 'col1[6]:1979-03-06 00:00:00 +0000 UTC '
 'col1[7]:1jU2JJcq0I1gGnOZm9ZtwUP9CIq5HEh+oG6alwM+0rSc5c6IQBvtXs8oAdo/8ZG1VXtvnTMbtHFiQRVyz3omv2cCmZE960ZzFK3mF34SbL+I7/VmhqhrSD1aKR7il2z6yUVHgkJTquiUVMNygNOqfkWvpFKToK8PJN9fUnNRlolmKiGiCYupFG4gb1jCuyiRxAYDqvefRZLvdyyu32VfPSCRFFydVo6UlDZch6qDsdA2zjV9PLotnGpYzKA9MzjShiNeVIIGrXYRoxJbDdsLrP3R12M16I67Atslc2YaWcJMfm3fFSZ94ZqjroI7H2ltkNo0un3PUAzdQpYCuZAz1j9UgCZevIUEmBOJdn45V5EJ8aXJgfxXWSDVIicuNT/i2YCqm0lPgDTqWhHHcSa86C9sjIeD2uT+ '
 'col1[8]:218.169.229.103 col1[9]:81.102.49.208 '
 'col2:U0NUdWVIyeYrAb3BlzvRMn6dUux9rii0UFEiH+LHzYP col3:927484090770182872 '
 'col4:map[5986897694310434673:RODStjHDVvOtzVGXtTOzcuYU33q5pVZULxoMmVPR7oPPFhEhKGGnh7t0DmmdoEE5/AUBwCb2rcF0aQXLDjnCZSj8pXzqanhCrZK4RSO+zcDRb7BfpeOU7JrEKJHjuj5YVmlmfMb4p5DX5s4qaWpNwL5Go51KUDGETzTXQ+Y3+vJ5m86UsycLd0vPSZJIUnTobMDs6glLaZ8m5/tFjtpc5RjAXtNgE08+HwINMv1V4OAUEe6ZSGmZnx2LqGNBnnMQUZjRRnZDvHJpxG3LOPYdaNYyjSOfxbqrGcAyU+Nag0Fl9sOEkDLwZL4D7/U6XKBbIaut86t5U+SoDz/sTne9ZXWD6F32A1AzUJYTvcmTaWf9scLVm908HB2MrYL2GA2iPIozNWqwWQeQtFEIkb3H172Tbqafva2yAp7VuswQVWGsIqXykagwAHMDw7voDYgZA6qvGWWoUnFzdsu8dCg39o8xw5ubVABNn0vflbkxzoF2R7CycJHfA] '
 'col5:4006430191304441957 col6:5313336783095743104 col7:2011-08-26 00:00:00 '
 '+0000 UTC col8:map[udt_3663736436_0:925b4980-d71b-11d1-9b73-02c6da6b609d '
 'udt_3663736436_1:[53 53 54 97 53 50 55 54 53 97 54 50 54 53 53 49 54 98 55 '
 '56 54 98 54 101 54 53 52 100 53 51 55 49 55 55 53 54 51 49 54 99 54 51 55 55 '
 '51 54 54 98 52 100 55 54 51 50 52 100 52 49 51 56 52 56 52 99 53 97 55 48 55 '
 '97 51 55 54 99 54 101 55 56 54 51 54 101 51 56 54 97 53 50 53 48 54 97 55 48 '
 '55 57 55 57 50 98 50 102 51 54 55 51 53 49 54 50 54 102 55 54 54 49 52 100 '
 '55 52 52 102 54 54 55 49 53 48 50 98 55 54 50 102 54 53 54 57 54 55 55 54 51 '
 '48 50 102 52 101 52 49 54 101 50 98 51 49 54 52 50 98 52 53 55 56 53 51 51 '
 '49 51 53 53 49 54 102 55 51 52 57 55 97 52 101 54 100 52 52 52 54 55 50 52 '
 '50 51 49 52 50 53 55 54 101 51 50 55 50 54 101 54 102 55 51 53 54 54 50 54 '
 '102 54 99 54 54 54 53 53 48 54 52 51 54 52 49 55 49 55 50 54 99 53 50 51 50 '
 '55 52 50 102 54 53 55 49 54 57 52 51 54 57 54 50 51 48 51 52 52 52 52 50 50 '
 '102 54 49 54 53 52 52 51 49 51 50 53 97 51 50 54 102 53 56 52 49 52 101 53 '
 '54 53 57 54 57 51 48 52 51 51 52 54 57 51 48 54 101 52 99 51 53 53 50 52 101 '
 '52 49 55 50 53 48 51 57 55 52 52 49 52 56 55 51 54 101 52 53 54 99 53 51 55 '
 '51 50 98 50 102 54 53 51 56 52 56 53 49 52 51 53 51 52 97 52 51 54 97 55 49 '
 '54 53 52 54 53 57 53 51 52 49 53 50 53 50 53 97 54 50 51 48 52 100 53 57 53 '
 '54 54 50 53 55 55 48 53 49 52 52 54 50 54 55 54 97 52 101 55 57 50 98 50 102 '
 '54 54 54 100 52 99 52 102 54 54 54 54 54 98 51 48 53 52 52 98] '
 'udt_3663736436_2:-69 udt_3663736436_3:false '
 'udt_3663736436_4:487d9400-d593-11cc-9b71-02c6da6b609d '
 'udt_3663736436_5:0.23399164 '
 'udt_3663736436_6:a5ef9580-6ab8-11bc-9b72-02c6da6b609d udt_3663736436_7:{0 0 '
 '2280000000000} '
 'udt_3663736436_8:RDu3MQ1H7B8VuWx3pkXYhxRC1n5KCobS/f1znvhTqF+W4C7+uPGTNQzIQHORCJqhQ5KjeWJv8LgD6JlxbKmnQQ/rS1sQWp9juCaQU+qIkj3od9NjeSgW2c6FPSkz9h7xyDi0ITVko2fwG9hHS9XSM1pJvAn+AZYVs7i6qXz1m9nBwubyztQrkimoUYsDGkJxngu4ucV2YQ5bTqiJrvyNvfIv7JpOAnZv6zZ8VNZec7DZPyUjP1mhIHDDiGSeTlxaRHBbyZh+muQztGyS6zTVOI] '
 'col9:0.2268903 pk0:18980 pk1:26005 pk2:-71]')

<<<<<<<

  • Restore Monitor Stack command: $ hydra investigate show-monitor c116e017-a6b7-49c4-a952-d7557c533c69
  • Restore monitor on AWS instance using Jenkins job
  • Show all stored logs command: $ hydra investigate show-logs c116e017-a6b7-49c4-a952-d7557c533c69

Logs:

Jenkins job URL
@fgelcer
Copy link
Contributor

fgelcer commented Nov 30, 2022

what is it failing for? results differ?

@fgelcer
Copy link
Contributor

fgelcer commented Dec 1, 2022

@aleksbykov , perhaps it would be fixed by backporting #5411 to this branch?

@roydahan @fruch ^^

@fgelcer
Copy link
Contributor

fgelcer commented Dec 1, 2022

could be related to scylladb/scylladb#11103

@KnifeyMoloko
Copy link
Contributor Author

Closing as duplicate of: #4692

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@KnifeyMoloko KnifeyMoloko

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@KnifeyMoloko @fgelcer