Atomic ingest #4895
Closed
Atomic ingest #4895
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
@riversand963 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
siying
reviewed
Jan 17, 2019
riversand963
force-pushed
the
atomic_ingest
branch
from
b242c33
to
04404c5
Compare
|
@riversand963 has updated the pull request. Re-import the pull request |
There was a problem hiding this comment.
@riversand963 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
riversand963
force-pushed
the
atomic_ingest
branch
from
04404c5
to
45d60e4
Compare
|
@riversand963 has updated the pull request. Re-import the pull request |
|
@riversand963 has updated the pull request. Re-import the pull request |
There was a problem hiding this comment.
@riversand963 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
@riversand963 has updated the pull request. Re-import the pull request |
1 similar comment
|
@riversand963 has updated the pull request. Re-import the pull request |
siying
reviewed
Jan 22, 2019
|
@riversand963 has updated the pull request. Re-import the pull request |
siying
approved these changes
Jan 24, 2019
riversand963
force-pushed
the
atomic_ingest
branch
from
a0c1e66
to
427c8c9
Compare
|
@riversand963 has updated the pull request. Re-import the pull request |
|
btw, I believe this PR can be merged only after #4922 is merged. Otherwise we will see error |
|
@riversand963 has updated the pull request. Re-import the pull request |
riversand963
force-pushed
the
atomic_ingest
branch
from
775af76
to
54f1150
Compare
|
@riversand963 has updated the pull request. Re-import the pull request |
riversand963
force-pushed
the
atomic_ingest
branch
from
54f1150
to
bd67672
Compare
|
@riversand963 has updated the pull request. Re-import the pull request |
There was a problem hiding this comment.
@riversand963 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
@riversand963 has updated the pull request. Re-import the pull request |
There was a problem hiding this comment.
@riversand963 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
@riversand963 has updated the pull request. Re-import the pull request |
inikep
pushed a commit
to inikep/percona-server
that referenced
this pull request
Aug 24, 2023
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
inikep
pushed a commit
to percona/percona-server
that referenced
this pull request
Sep 15, 2023
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
inikep
pushed a commit
to inikep/percona-server
that referenced
this pull request
Oct 16, 2023
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
inikep
pushed a commit
to inikep/percona-server
that referenced
this pull request
Oct 25, 2023
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
inikep
pushed a commit
to inikep/percona-server
that referenced
this pull request
Oct 26, 2023
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
inikep
pushed a commit
to inikep/percona-server
that referenced
this pull request
Oct 26, 2023
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Nov 1, 2023
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Nov 10, 2023
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
inikep
pushed a commit
to inikep/percona-server
that referenced
this pull request
Jan 17, 2024
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
inikep
pushed a commit
to inikep/percona-server
that referenced
this pull request
Jan 17, 2024
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Jan 18, 2024
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Jan 21, 2024
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Jan 26, 2024
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Jan 30, 2024
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
inikep
pushed a commit
to inikep/percona-server
that referenced
this pull request
Apr 9, 2024
Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. @update-submodule: rocksdb
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Apr 12, 2024
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Apr 12, 2024
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Apr 12, 2024
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Apr 15, 2024
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Apr 15, 2024
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Apr 15, 2024
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Apr 16, 2024
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Apr 16, 2024
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Apr 16, 2024
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Apr 16, 2024
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Apr 16, 2024
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Apr 16, 2024
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Apr 18, 2024
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Apr 29, 2024
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
May 13, 2024
Upstream commit ID : fb-mysql-5.6.35/269d05b0beb761d9f9ba057f12b62b0077a3c4a2 PS-6864 : Merge fb-prod201902 Summary: Enable atomic bulk loading - bulk loading using SST APIs are now atomic. You should not observe any in between state if there is any failures in ingestion (there are some caveats, as usual, that needs to be investigated/documented). This change essentially delays all SST ingestion until finish_bulk_load, and ingest them all in one batch. Care is taken to: 1. Make sure the state management of multiple `Rdb_sst_info` are done transactionally as well to match the ingestion, 2. Works well with closing connections and ALTER TABLE statement race conditions (however under race conditions the ingestion are forcefully interrupted and therefore no longer truly atomic - this needs to be investigated to see if there are other approaches). 3. Files are properly cleaned up in the failure case NOTE: I specifically avoided doing refactoring in order to make code review easier. Finish_bulk_load is becoming too big and that will get addressed later. This changes takes advantage of latest [PR][(facebook/rocksdb#4895) from Yanqin Jin which supports ingesting SST from multiple column families. NOTE: This change updates rocksdb to latest commit in the PR and will update to the proper rocksdb version once the PR is merged and we've decided on a proper RocksDB version to consume. update-submodule: rocksdb fbshipit-source-id: 767c36232b6
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Make file ingestion atomic.
Summary: as title.
Ingesting external SST files into multiple column families should be atomic. If
a crash occurs and db reopens, either all column families have successfully
ingested the files before the crash, or non of the ingestions have any effect
on the state of the db.
Also add unit tests for atomic ingestion.
Note that the unit test here does not cover the case of incomplete atomic group
in the MANIFEST, which is covered in VersionSetTest already.
Test Plan: