Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multi-DC writes may fail during schema change #3393

Closed
tgrabiec opened this issue Apr 25, 2018 · 0 comments
Closed

Multi-DC writes may fail during schema change #3393

tgrabiec opened this issue Apr 25, 2018 · 0 comments
Assignees
@tgrabiec
Labels
type/bug

Comments

@tgrabiec
Copy link
Contributor

Currently, replicas request schema definition from the local coordinator, not from the original coordinator in another DC. If the local coordinator hasn't yet learned about the new schema version used by the original coordinator, replication may fail with the following message:

  storage_proxy - Failed to apply mutation from 1.1.1.1#5: std::_Nested_exception<schema_version_loading_failed> (Failed to load schema version 32893223-a911-3a01-ad70-df1eb2a15db1): std::runtime_error (Schema version 32893223-a911-3a01-ad70-df1eb2a15db1 not found)
@tgrabiec tgrabiec self-assigned this Apr 25, 2018
@tgrabiec tgrabiec mentioned this issue Apr 25, 2018
Closed
avikivity pushed a commit that referenced this issue May 7, 2018
@tgrabiec @avikivity
storage_proxy: Request schema from the coordinator in the original DC
202b4e6 
The mutation forwarding intermediary (src_addr) may not always know
about the schema which was used by the original coordinator. I think
this may be the cause of the "Schema version ... not found" error seen
in one of the clusters which entered some pathological state:

  storage_proxy - Failed to apply mutation from 1.1.1.1#5: std::_Nested_exception<schema_version_loading_failed> (Failed to load schema version 32893223-a911-3a01-ad70-df1eb2a15db1): std::runtime_error (Schema version 32893223-a911-3a01-ad70-df1eb2a15db1 not found)

Fixes #3393.

Message-Id: <1524639030-1696-1-git-send-email-tgrabiec@scylladb.com>
(cherry picked from commit 423712f)
denesb pushed a commit to denesb/scylla that referenced this issue Jul 2, 2018
@tgrabiec @avikivity
storage_proxy: Request schema from the coordinator in the original DC
0e34c8b 
The mutation forwarding intermediary (src_addr) may not always know
about the schema which was used by the original coordinator. I think
this may be the cause of the "Schema version ... not found" error seen
in one of the clusters which entered some pathological state:

  storage_proxy - Failed to apply mutation from 1.1.1.1#5: std::_Nested_exception<schema_version_loading_failed> (Failed to load schema version 32893223-a911-3a01-ad70-df1eb2a15db1): std::runtime_error (Schema version 32893223-a911-3a01-ad70-df1eb2a15db1 not found)

Fixes scylladb#3393.

Message-Id: <1524639030-1696-1-git-send-email-tgrabiec@scylladb.com>
(cherry picked from commit 423712f)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tgrabiec tgrabiec

Labels
type/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tgrabiec