Add Add Operator::initialize API to handle customized operation init #6422
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for meta-velox canceled.
|
facebook-github-bot
added
the
CLA Signed
mbasmanova
reviewed
Sep 5, 2023
velox/exec/Operator.h
Outdated
| @@ -301,69 +301,75 @@ class Operator : public BaseRuntimeStatWriter { | |||
|
|
|||
| virtual ~Operator() = default; | |||
|
|
|||
| // Returns true if 'this' can accept input. Not used if operator is a source | |||
| // operator, e.g. the first operator in the pipeline. | |||
| virtual void init(); | |||
There was a problem hiding this comment.
Maybe "initialize".
CC: @aditi-pandit
mbasmanova
reviewed
Sep 5, 2023
mbasmanova
reviewed
Sep 5, 2023
mbasmanova
reviewed
Sep 5, 2023
mbasmanova
reviewed
Sep 5, 2023
velox/exec/Operator.h
Outdated
| @@ -301,69 +301,75 @@ class Operator : public BaseRuntimeStatWriter { | |||
|
|
|||
| virtual ~Operator() = default; | |||
|
|
|||
| // Returns true if 'this' can accept input. Not used if operator is a source | |||
| // operator, e.g. the first operator in the pipeline. | |||
| virtual void init(); | |||
There was a problem hiding this comment.
Let's add documentation to explain that this method is for doing any construction work that requires memory allocations. Let's also comment that Operator's constructor should not allocate memory.
xiaoxmeng
force-pushed
the
init
branch
4 times, most recently
from
3fc06ac
to
f5ee073
Compare
|
@xiaoxmeng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
mbasmanova
approved these changes
Sep 5, 2023
xiaoxmeng
changed the title
|
@xiaoxmeng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
@mbasmanova thanks for the quick review! |
|
@xiaoxmeng merged this pull request in 8f6be9d. |
|
Conbench analyzed the 1 benchmark run on commit There were no benchmark performance regressions. 🎉 The full Conbench report has more details. |
codyschierbeck
pushed a commit
to codyschierbeck/velox
that referenced
this pull request
Sep 27, 2023
…acebookincubator#6422) Summary: Add operator init API to handle heavy customized operator initialization work such as memory pool allocations. The operator init is called by driver operator init method when the driver starts execution for the first time. This is required as memory allocation can trigger memory arbitration which might grab the task lock for memory reclamation. This can cause deadlock as the driver creation is under the lock (see [PR](facebookincubator#6395) for details. Later on, we will add memory usage check after driver creation to enforce there is no memory allocations from memory pool on driver creation under the task lock. Pull Request resolved: facebookincubator#6422 Reviewed By: mbasmanova Differential Revision: D48979505 Pulled By: xiaoxmeng fbshipit-source-id: 68748ef179633c0ef6c99a5ef95fd99c2313d81d
codyschierbeck
pushed a commit
to codyschierbeck/velox
that referenced
this pull request
Sep 27, 2023
…acebookincubator#6422) Summary: Add operator init API to handle heavy customized operator initialization work such as memory pool allocations. The operator init is called by driver operator init method when the driver starts execution for the first time. This is required as memory allocation can trigger memory arbitration which might grab the task lock for memory reclamation. This can cause deadlock as the driver creation is under the lock (see [PR](facebookincubator#6395) for details. Later on, we will add memory usage check after driver creation to enforce there is no memory allocations from memory pool on driver creation under the task lock. Pull Request resolved: facebookincubator#6422 Reviewed By: mbasmanova Differential Revision: D48979505 Pulled By: xiaoxmeng fbshipit-source-id: 68748ef179633c0ef6c99a5ef95fd99c2313d81d
codyschierbeck
pushed a commit
to codyschierbeck/velox
that referenced
this pull request
Sep 27, 2023
…acebookincubator#6422) Summary: Add operator init API to handle heavy customized operator initialization work such as memory pool allocations. The operator init is called by driver operator init method when the driver starts execution for the first time. This is required as memory allocation can trigger memory arbitration which might grab the task lock for memory reclamation. This can cause deadlock as the driver creation is under the lock (see [PR](facebookincubator#6395) for details. Later on, we will add memory usage check after driver creation to enforce there is no memory allocations from memory pool on driver creation under the task lock. Pull Request resolved: facebookincubator#6422 Reviewed By: mbasmanova Differential Revision: D48979505 Pulled By: xiaoxmeng fbshipit-source-id: 68748ef179633c0ef6c99a5ef95fd99c2313d81d
ericyuliu
pushed a commit
to ericyuliu/velox
that referenced
this pull request
Oct 12, 2023
…acebookincubator#6422) Summary: Add operator init API to handle heavy customized operator initialization work such as memory pool allocations. The operator init is called by driver operator init method when the driver starts execution for the first time. This is required as memory allocation can trigger memory arbitration which might grab the task lock for memory reclamation. This can cause deadlock as the driver creation is under the lock (see [PR](facebookincubator#6395) for details. Later on, we will add memory usage check after driver creation to enforce there is no memory allocations from memory pool on driver creation under the task lock. Pull Request resolved: facebookincubator#6422 Reviewed By: mbasmanova Differential Revision: D48979505 Pulled By: xiaoxmeng fbshipit-source-id: 68748ef179633c0ef6c99a5ef95fd99c2313d81d
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add operator init API to handle heavy customized operator initialization
work such as memory pool allocations. The operator init is called by
driver operator init method when the driver starts execution for the first
time. This is required as memory allocation can trigger memory arbitration
which might grab the task lock for memory reclamation. This can cause
deadlock as the driver creation is under the lock (see PR for details.
Later on, we will add memory usage check after driver creation to enforce
there is no memory allocations from memory pool on driver creation under
the task lock.