Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: Configure Ray plasma storage #66

Merged
merged 1 commit into from Mar 2, 2023
Merged

chore: Configure Ray plasma storage #66

merged 1 commit into from Mar 2, 2023

Conversation

jamesclark-Zapata
Copy link
Contributor

@jamesclark-Zapata jamesclark-Zapata commented Mar 2, 2023 •
edited

The problem

Ray's plasma directory needs to be configured in Portal

This PR's solution

  • Configures Ray's plasma directory
  • Allows Portal to override this location

Checklist

Check that this PR satisfies the following items:

  • Tests have been added for new features/changed behavior (if no new features have been added, check the box).
  • The changelog file has been updated with a user-readable description of the changes (if the change isn't visible to the user in any way, check the box).
  • The PR's title is prefixed with <feat/fix/chore/internal/docs>[!]:

@jamesclark-Zapata jamesclark-Zapata force-pushed the chore/jamesclark-zapata/add-plasma-dir-ray branch from 70cec8a to 3648b76 Compare March 2, 2023 16:15
@jamesclark-Zapata jamesclark-Zapata marked this pull request as ready for review March 2, 2023 16:15
@jamesclark-Zapata jamesclark-Zapata requested a review from a team as a code owner March 2, 2023 16:15
@jamesclark-Zapata jamesclark-Zapata force-pushed the chore/jamesclark-zapata/add-plasma-dir-ray branch from 3648b76 to e23e310 Compare March 2, 2023 16:18
@github-actions
Copy link

github-actions bot commented Mar 2, 2023 •
edited

🚀 Code Coverage

You are using PYTHON: /D/a/orquestra-workflow-sdk/orquestra-workflow-sdk/my_little_venv/Scripts/python.exe
C:/Program FilesPython 3.9.13
Repository: https://github.com/zapatacomputing/orquestra-workflow-sdk
Python Modules Covered: src.orquestra/sdk/examples src.orquestra/sdk/schema src.orquestra/sdk/secrets src.orquestra/sdk/v2 src.orquestra/sdk/_base src.orquestra/sdk/_ray
-------------------------------------------------------------------------------
Name                                                                    Stmts   Miss  Cover   Missing
-----------------------------------------------------------------------------------------------------
src\orquestra\sdk\_base\_api\_config.py                                   196     42    79%   259-278, 363-371, 373, 445, 467-560, 610-616
src\orquestra\sdk\_base\_api\_task_run.py                                  96      0   100%
src\orquestra\sdk\_base\_api\_wf_run.py                                   167     11    93%   99-100, 113-114, 168-177, 338-339
src\orquestra\sdk\_base\_ast.py                                           108      1    99%   222
src\orquestra\sdk\_base\_config.py                                        187      0   100%
src\orquestra\sdk\_base\_conversions\_ids.py                               38      0   100%
src\orquestra\sdk\_base\_conversions\_imports.py                           60      0   100%
src\orquestra\sdk\_base\_conversions\_invocations.py                      104      0   100%
src\orquestra\sdk\_base\_conversions\_yaml_exporter.py                     38      1    97%   39
src\orquestra\sdk\_base\_db\_db.py                                         67      0   100%
src\orquestra\sdk\_base\_db\_migration.py                                  15      0   100%
src\orquestra\sdk\_base\_driver\_ce_runtime.py                            100      0   100%
src\orquestra\sdk\_base\_driver\_client.py                                167      2    99%   233-238
src\orquestra\sdk\_base\_driver\_exceptions.py                             54      0   100%
src\orquestra\sdk\_base\_driver\_models.py                                 94      0   100%
src\orquestra\sdk\_base\_dsl.py                                           354      5    99%   39, 747, 833, 846, 860
src\orquestra\sdk\_base\_env.py                                             8      0   100%
src\orquestra\sdk\_base\_exec_ctx.py                                       25      0   100%
src\orquestra\sdk\_base\_factory.py                                        17      4    76%   34-39
src\orquestra\sdk\_base\_graphs.py                                         43      1    98%   65
src\orquestra\sdk\_base\_in_process_runtime.py                            111      8    93%   62-65, 119-120, 171, 231
src\orquestra\sdk\_base\_log_adapter.py                                    57      9    84%   86-88, 165-176, 183-184
src\orquestra\sdk\_base\_qe\_client.py                                     60      2    97%   108-109
src\orquestra\sdk\_base\_qe\_qe_runtime.py                                283     13    95%   91-92, 127, 295-296, 301, 436, 490-491, 574, 617, 693, 823
src\orquestra\sdk\_base\_services.py                                       55      4    93%   22, 25, 28, 31
src\orquestra\sdk\_base\_testing\_connections.py                           38      5    87%   35-36, 79-81
src\orquestra\sdk\_base\_testing\_example_wfs.py                          162     37    77%   13, 34, 44, 49, 54-57, 62-65, 85, 96, 108, 114-115, 177, 182-184, 189-193, 202-210, 237-239, 265-269, 279-282, 292
src\orquestra\sdk\_base\_traversal.py                                     242      3    99%   234, 478, 498
src\orquestra\sdk\_base\_viz.py                                            58      3    95%   63, 68, 127
src\orquestra\sdk\_base\_workflow.py                                      198     12    94%   169, 181, 274, 404-408, 456-457, 475, 492, 501
src\orquestra\sdk\_base\abc.py                                             44      2    95%   44, 59
src\orquestra\sdk\_base\cli\_corq\_format\color.py                          6      0   100%
src\orquestra\sdk\_base\cli\_corq\_format\per_command.py                  107     13    88%   45, 55, 73-86, 179-184, 192
src\orquestra\sdk\_base\cli\_corq\_main.py                                137      4    97%   531-533, 541
src\orquestra\sdk\_base\cli\_corq\action\_exception_handling.py           105      9    91%   77-78, 101, 291, 364, 394, 409, 453, 467
src\orquestra\sdk\_base\cli\_corq\action\_get_artifacts.py                 20      0   100%
src\orquestra\sdk\_base\cli\_corq\action\_get_default_config.py             7      0   100%
src\orquestra\sdk\_base\cli\_corq\action\_get_logs.py                      17      0   100%
src\orquestra\sdk\_base\cli\_corq\action\_get_task_def.py                  23      4    83%   28-31
src\orquestra\sdk\_base\cli\_corq\action\_get_workflow_def.py              22      6    73%   28-33, 43-47
src\orquestra\sdk\_base\cli\_corq\action\_get_workflow_run.py              29      1    97%   67
src\orquestra\sdk\_base\cli\_corq\action\_get_workflow_run_results.py      23      0   100%
src\orquestra\sdk\_base\cli\_corq\action\_list_workflow_runs.py            42      0   100%
src\orquestra\sdk\_base\cli\_corq\action\_login.py                         85      3    96%   73, 241-245
src\orquestra\sdk\_base\cli\_corq\action\_set_default_config.py             7      0   100%
src\orquestra\sdk\_base\cli\_corq\action\_set_token.py                     23      2    91%   30-31
src\orquestra\sdk\_base\cli\_corq\action\_stop_workflow_run.py             14      0   100%
src\orquestra\sdk\_base\cli\_corq\action\_submit_workflow_def.py           43      1    98%   81
src\orquestra\sdk\_base\cli\_dorq\_arg_resolvers.py                       132      1    99%   65
src\orquestra\sdk\_base\cli\_dorq\_dumpers.py                              26      0   100%
src\orquestra\sdk\_base\cli\_dorq\_entry.py                               117     33    72%   99-102, 113-116, 141-144, 164-167, 182-185, 269-272, 286-289, 307-310, 325-328, 338-341, 368-371
src\orquestra\sdk\_base\cli\_dorq\_login\_login.py                         44      2    95%   44-45
src\orquestra\sdk\_base\cli\_dorq\_login\_login_server.py                  31      9    71%   16-23, 26
src\orquestra\sdk\_base\cli\_dorq\_repos.py                               193      4    98%   200-201, 331-332
src\orquestra\sdk\_base\cli\_dorq\_task\_logs.py                           29      2    93%   52-53
src\orquestra\sdk\_base\cli\_dorq\_task\_results.py                        31      2    94%   56-57
src\orquestra\sdk\_base\cli\_dorq\_ui\_errors.py                           57      4    93%   25-31, 108
src\orquestra\sdk\_base\cli\_dorq\_ui\_models.py                           20      0   100%
src\orquestra\sdk\_base\cli\_dorq\_ui\_presenters.py                       96      4    96%   41-49, 165-172
src\orquestra\sdk\_base\cli\_dorq\_ui\_prompts.py                          70     47    33%   42, 51, 78-91, 107-116, 125, 134, 161-175, 201-232, 255-271
src\orquestra\sdk\_base\cli\_dorq\_workflow\_list.py                       26      2    92%   61-62
src\orquestra\sdk\_base\cli\_dorq\_workflow\_logs.py                       27      2    93%   57-58
src\orquestra\sdk\_base\cli\_dorq\_workflow\_results.py                    29      0   100%
src\orquestra\sdk\_base\cli\_dorq\_workflow\_stop.py                       26      0   100%
src\orquestra\sdk\_base\cli\_dorq\_workflow\_submit.py                     41      1    98%   117
src\orquestra\sdk\_base\cli\_dorq\_workflow\_view.py                       23      2    91%   47-48
src\orquestra\sdk\_base\dispatch.py                                       106      2    98%   68-71
src\orquestra\sdk\_base\loader.py                                          63      0   100%
src\orquestra\sdk\_base\serde.py                                           75      0   100%
src\orquestra\sdk\_ray\_client.py                                          61     10    84%   18-24, 95-100, 113, 169
src\orquestra\sdk\_ray\_dag.py                                            331     94    72%   49-56, 154, 183-206, 276-284, 294-315, 324-344, 348-452, 459, 486-489, 502, 685-700, 722-728, 757-758, 986-1000
src\orquestra\sdk\_ray\_id_gen.py                                           9      0   100%
src\orquestra\sdk\_ray\_ray_logs.py                                        66      0   100%
src\orquestra\sdk\examples\exportable_wf.py                                22      3    86%   30, 41, 58
src\orquestra\sdk\examples\workflow_defs.py                                21     10    52%   11-14, 19, 24-28
src\orquestra\sdk\exceptions.py                                            86      1    99%   170
src\orquestra\sdk\packaging.py                                             24      0   100%
src\orquestra\sdk\schema\configs.py                                        18      0   100%
src\orquestra\sdk\schema\ir.py                                            120      0   100%
src\orquestra\sdk\schema\local_database.py                                  7      0   100%
src\orquestra\sdk\schema\responses.py                                      99      0   100%
src\orquestra\sdk\schema\workflow_run.py                                   31      0   100%
src\orquestra\sdk\schema\yaml_model.py                                     63      0   100%
src\orquestra\sdk\secrets\_api.py                                          50      6    88%   55, 57, 89, 126, 161, 163
src\orquestra\sdk\secrets\_auth.py                                         33      0   100%
src\orquestra\sdk\secrets\_client.py                                       58      1    98%   37
src\orquestra\sdk\secrets\_exceptions.py                                   15      2    87%   44-45
src\orquestra\sdk\secrets\_models.py                                       11      0   100%
-----------------------------------------------------------------------------------------------------
TOTAL                                                                    6343    452    93%
-------------
Diff Coverage
Diff: origin/main...HEAD, staged and unstaged changes
-------------
src/orquestra/sdk/_base/_env.py (100%)
src/orquestra/sdk/_base/_services.py (100%)
-------------
Total:   7 lines
Missing: 0 lines
Coverage: 100%
-------------

@jamesclark-Zapata jamesclark-Zapata merged commit a2255ad into main Mar 2, 2023
@jamesclark-Zapata jamesclark-Zapata deleted the chore/jamesclark-zapata/add-plasma-dir-ray branch March 2, 2023 16:52
@jamesclark-Zapata jamesclark-Zapata mentioned this pull request Mar 6, 2023
Merged
3 tasks
jamesclark-Zapata added a commit that referenced this pull request Mar 6, 2023
@jamesclark-Zapata
fix: Make Ray dirs before starting Ray (#70)
131923d 
# The problem

Ray was failing to start after #66 because the directory it set may not
exist

# This PR's solution

Ensures all Ray dirs are made before starting Ray

There's an issue with the test on CI, see ORQSDK-771.

# Checklist

_Check that this PR satisfies the following items:_

- [x] Tests have been added for new features/changed behavior (if no new
features have been added, check the box).
- [x] The [changelog file](CHANGELOG.md) has been updated with a
user-readable description of the changes (if the change isn't visible to
the user in any way, check the box).
- [x] The PR's title is prefixed with
`<feat/fix/chore/internal/docs>[!]:`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alexjuda alexjuda alexjuda approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@jamesclark-Zapata @alexjuda