[Proposal] Implement Scorll Parallel Retrieve Cursor. #120
Unanswered
wenchaozhang-123
asked this question in
Proposal
Replies: 1 comment
-
Hi @wenchaozhang-123 thanks for your proposal. Your proposal is identified as No.3 and accessible via https://github.com/cloudberrydb/community/blob/main/proposals/cp-3/cp-3.md. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Proposers
@wenchaozhang-123
Proposal Status
Under Discussion
Abstract
To support Ray using data in CBDB to finish ML work, we should implement a new cursor named scroll parallel retrieve cursor which supports rescan on each segment. This proposal describes the problems in parallel retrieve cursor and the main idea of how to enhance it.
Motivation
Now, the endpoint of parallel retrieve cursor will be disconnected after retrieves all data in segment. What's more, it can't support the rescan of the table. This is unfriendly to Ray ML work. Hence, we expect to implement a new parallel retrieve cursor, which can support rescan table in segment.
Implementation
Design and problem of parallel retrieve cursor:
![parallel retrieve cursor](https://private-user-images.githubusercontent.com/53178068/258387172-8a43c9b3-f790-4e8a-a23d-49c8c14ec46e.jpg?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU2MjQ2NjIsIm5iZiI6MTcyNTYyNDM2MiwicGF0aCI6Ii81MzE3ODA2OC8yNTgzODcxNzItOGE0M2M5YjMtZjc5MC00ZThhLWEyM2QtNDljOGMxNGVjNDZlLmpwZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTA2VDEyMDYwMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTZiYmIxYTMzY2UzNzQzMjJlYTk0YjBiNGQ4ZTk4YTNkMmVmYzk2MjBjOWUxZTg0YzQwNjc2NzVjOTZjYmZiMzQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.amNiJi-zWOCH_NEfFXmRpo_wg6pDTvECUysGOXnlzEc)
Every parallel retrieve cursor includes one or more endpoints, each endpoint can read tuples. After retrieves all tuples in one endpoint, the connection will be released. We expect Ray to use the parallel retrieve cursor, which needs to rescan the table in one endpoint. However, as we describe above, parallel retrieve cursor doesn't support that. The design of parallel retrieve cursor as follows:
As a result, we should hold connection after retrieves cursor. What's more, the tuple should be rescanned in the endpoint. To achieve that, we should implement a new cursor which can support rescan the table in the endpoint. The main idea is that we should hold the endpoint connection until SQL command to close it. And we should store the tuple retrieved from the table to the tuplestore, which support the execution of the rescan. The main execution flow is as follows:
![scorll parallel retrieve cursor](https://private-user-images.githubusercontent.com/53178068/258387221-beb48867-250c-4b07-8381-08ffd50d0340.jpg?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU2MjQ2NjIsIm5iZiI6MTcyNTYyNDM2MiwicGF0aCI6Ii81MzE3ODA2OC8yNTgzODcyMjEtYmViNDg4NjctMjUwYy00YjA3LTgzODEtMDhmZmQ1MGQwMzQwLmpwZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTA2VDEyMDYwMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTNiNDJmOTc3ODQwNGQyZDg1ZWU4ZTk2OGNiMDYzY2ViZGY1MzRmZDcxZDU1YzZjNTAxZDZhODllZDEyZmFkODEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Fb2Qrg-rRMncPoDb_kg-ybWrUUX6nZ756yG0Bq7oyHs)
#104
Rollout/Adoption Plan
No response
Are you willing to submit a PR?
Beta Was this translation helpful? Give feedback.
All reactions