Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test_tools.TestScyllaSsstableSchemaLoading.test_table_dir_data_dir is flaky #13553

Closed
denesb opened this issue Apr 18, 2023 · 5 comments
Closed

test_tools.TestScyllaSsstableSchemaLoading.test_table_dir_data_dir is flaky #13553

denesb opened this issue Apr 18, 2023 · 5 comments
Assignees
@denesb
Labels
symptom/ci stability Issues that failed in ScyllaDB CI - tests and framework
Milestone
5.3

Comments

@denesb
Copy link
Contributor

denesb commented Apr 18, 2023

Seen in:

14:40:01  DEBUG 2023-04-17 13:01:23,885 [shard 0] scylla-sstable - Single schema source provided
14:40:01  ERROR 2023-04-17 13:01:24,315 [shard 0] sstable - Could not create SSTable component /jenkins/workspace/scylla-master/next/scylla/testlog/x86_64/debug/scylla-10/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/me-58-big-TOC.txt. Found exception: std::filesystem::__cxx11::filesystem_error (error system:2, filesystem error: open failed: No such file or directory [/jenkins/workspace/scylla-master/next/scylla/testlog/x86_64/debug/scylla-10/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/me-58-big-TOC.txt])
14:40:01  error: could not load schema via schema-tables: std::system_error (error system:2, stat failed: No such file or directory)
14:40:01  FWARNING: debug mode. Not for benchmarking or production
@denesb denesb added the symptom/ci stability Issues that failed in ScyllaDB CI - tests and framework label Apr 18, 2023
@denesb denesb self-assigned this Apr 18, 2023
@denesb
Copy link
Contributor Author

denesb commented Apr 18, 2023 

[2023-04-18T01:25:53.852Z] DEBUG 2023-04-18 02:52:41,455 [shard 0] scylla-sstable - Single schema source provided
[2023-04-18T01:25:53.852Z] ERROR 2023-04-18 02:52:41,582 [shard 0] sstable - Could not create SSTable component /jenkins/workspace/scylla-master/next/scylla/testlog/x86_64/debug/scylla-10/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/me-164-big-Filter.db. Found exception: std::filesystem::__cxx11::filesystem_error (error system:2, filesystem error: open failed: No such file or directory [/jenkins/workspace/scylla-master/next/scylla/testlog/x86_64/debug/scylla-10/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/me-164-big-Filter.db])
[2023-04-18T01:25:53.852Z] ERROR 2023-04-18 02:52:41,583 [shard 0] sstable - Could not create SSTable component /jenkins/workspace/scylla-master/next/scylla/testlog/x86_64/debug/scylla-10/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/me-164-big-CompressionInfo.db. Found exception: std::filesystem::__cxx11::filesystem_error (error system:2, filesystem error: open failed: No such file or directory [/jenkins/workspace/scylla-master/next/scylla/testlog/x86_64/debug/scylla-10/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/me-164-big-CompressionInfo.db])
[2023-04-18T01:25:53.853Z] ERROR 2023-04-18 02:52:41,584 [shard 0] sstable - Could not create SSTable component /jenkins/workspace/scylla-master/next/scylla/testlog/x86_64/debug/scylla-10/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/me-164-big-Summary.db. Found exception: std::filesystem::__cxx11::filesystem_error (error system:2, filesystem error: open failed: No such file or directory [/jenkins/workspace/scylla-master/next/scylla/testlog/x86_64/debug/scylla-10/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/me-164-big-Summary.db])
[2023-04-18T01:25:53.853Z] ERROR 2023-04-18 02:52:41,585 [shard 0] sstable - Could not create SSTable component /jenkins/workspace/scylla-master/next/scylla/testlog/x86_64/debug/scylla-10/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/me-164-big-Index.db. Found exception: std::filesystem::__cxx11::filesystem_error (error system:2, filesystem error: open failed: No such file or directory [/jenkins/workspace/scylla-master/next/scylla/testlog/x86_64/debug/scylla-10/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/me-164-big-Index.db])
[2023-04-18T01:25:53.853Z] error: could not load schema via schema-tables: std::filesystem::__cxx11::filesystem_error (error system:2, filesystem error: open failed: No such file or directory [/jenkins/workspace/scylla-master/next/scylla/testlog/x86_64/debug/scylla-10/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/me-164-big-Index.db])

denesb added a commit to denesb/scylla that referenced this issue Apr 18, 2023
@denesb
test/cql-pytest: mark test_tools.TestScyllaSsstableSchemaLoading.test…
7bedddc 
…_table_dir_data_dir as xfail

This test fails from time-to-time. Until the investigation is ongoing,
mark it as xfail so it doesn't affext our workflows.

Refs: scylladb#13553
@denesb denesb mentioned this issue Apr 18, 2023
Closed
@denesb
Copy link
Contributor Author

denesb commented Apr 18, 2023

The test that fails just happens to be the first test running that accesses the data directory. There might be a compaction still running at that time, causing the disappearing sstables.

@denesb
Copy link
Contributor Author

denesb commented Apr 18, 2023

Note that later tests accessing the data dir have no problem.

denesb added a commit to denesb/scylla that referenced this issue Apr 18, 2023
@denesb
test/cql-pytest: TestScyllaSsstableSchemaLoading: wait for ongoing co…
3bd4b36 
…mpactions

The tests in this class rely on no compaction overlapping with their
progress. Despite disabling auto-compaction for the scope of this test
class, we still observed the very first test in the suite failing
sometimes in debug mode. This is most likely a compaction started before
disabling auto-compaction, which (due to the slow debug build) finishes
such that it deletes the compacted sstables, right as one of the tests
runs.
To prevent this, use the recently introduced tasks api to check and wait
for any ongoing compaction before proceeding with the tests.

Fixes: scylladb#13553
This was referenced Apr 18, 2023
Closed
Closed
@denesb
Copy link
Contributor Author

denesb commented Apr 18, 2023

After further investigation with the help of @bhalevy, the problem turns out to be that nodetool.no_autocompaction_context used the wrong api endpoints and dropped errors about it.

denesb added a commit to denesb/scylla that referenced this issue Apr 18, 2023
@denesb
test/cql-pytest/nodetool.py: no_autocompaction_context: use the corre…
e73e41e 
…ct API

This `with` context is supposed to disable, then re-enable
autocompaction for the given keyspaces, but it used the wrong API for
it, it used the column_family/autocompaction API, which operates on
column families, not keyspaces. This oversight led to a silent failure
because the code didn't check the result of the request.
Both are fixed in this patch:
* switch to use `storage_service/auto_compaction/{keyspace}` endpoint
* check the result of the API calls and report errors as exceptions

Fixes: scylladb#13553
denesb added a commit to denesb/scylla that referenced this issue Apr 18, 2023
@denesb
test/cql-pytest/nodetool.py: no_autocompaction_context: use the corre…
b23a7c0 
…ct API

This `with` context is supposed to disable, then re-enable
autocompaction for the given keyspaces, but it used the wrong API for
it, it used the column_family/autocompaction API, which operates on
column families, not keyspaces. This oversight led to a silent failure
because the code didn't check the result of the request.
Both are fixed in this patch:
* switch to use `storage_service/auto_compaction/{keyspace}` enpoint
* check the result of the API calls and report errors as exceptions

Fixes: scylladb#13553
@denesb denesb mentioned this issue Apr 18, 2023
Closed
@mykaul mykaul added this to the 5.3 milestone Apr 20, 2023
@avikivity
Copy link
Member

No vulnerable branches, not backporting.

denesb added a commit that referenced this issue Nov 7, 2023
@denesb
test/cql-pytest/nodetool.py: no_autocompaction_context: use the corre…
17e4d53 
…ct API

This `with` context is supposed to disable, then re-enable
autocompaction for the given keyspaces, but it used the wrong API for
it, it used the column_family/autocompaction API, which operates on
column families, not keyspaces. This oversight led to a silent failure
because the code didn't check the result of the request.
Both are fixed in this patch:
* switch to use `storage_service/auto_compaction/{keyspace}` endpoint
* check the result of the API calls and report errors as exceptions

Fixes: #13553

Closes #13568

(cherry picked from commit 66ee736)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@denesb denesb

Labels
symptom/ci stability Issues that failed in ScyllaDB CI - tests and framework
Projects
None yet
Milestone
5.3
4 participants
@avikivity @denesb @mykaul @scylladb-promoter