-
Notifications
You must be signed in to change notification settings - Fork 5.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Data] Introduce
concurrency
argument to replace ComputeStrategy in…
… map-like APIs (#41461) Generated doc for review - https://anyscale-ray--41461.com.readthedocs.build/en/41461/data/transforming-data.html#transforming-with-python-class . This PR is to add an extra `concurrency` argument into all map-like APIs (`map_batches`, `map`, `filter`, `flat_map`, `add_column`, `drop_columns`, `select_columns`), with the motivation to deprecate `compute` argument. The typing for new `concurrency` is ``` Optional[Union[int, Tuple[int, int]]] ``` So it allows user to set a fixed-sized actors pool, or an auto-scaling actors pool. For 2.9, the `compute` argument would still work, but will print out a warning message for users to migrate to use `concurrency`. So this PR does not break any existing code and maintains backward compatibility. Several other alternatives: * Use two arguments `min_concurrency`, `max_concurrency`: `max_concurrency` is already a reserved parameter for Ray Core. This represents the number of concurrent actor tasks in Ray Core. So this would introduce extra confusion for users. In addition, we are recommending our users to use a fixed-sized actors pool for now. These two arguments are only useful for auto-scaling actors pool. * Introduce a class like `ConcurrencyOption`: Do not see a need right now, and it would go back to have same issue with `ActorPoolStrategy`. We can always overload the type of `concurrency` and add more new types later, without breaking backward compatibility. * Overload the type of existing argument `compute`: This would also work and requires minimal change from user side. The naming of `compute` is more vague than `concurrency` though. Signed-off-by: Cheng Su <scnju13@gmail.com> Signed-off-by: Stephanie Wang <swang@cs.berkeley.edu> Co-authored-by: Stephanie Wang <swang@cs.berkeley.edu>
- Loading branch information
1 parent
19bedd1
commit c71f43c
Showing
15 changed files
with
465 additions
and
273 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.