Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support multiple containers per executor #5

Closed
yadudoc opened this issue Jun 13, 2019 · 0 comments
Closed

Support multiple containers per executor #5

yadudoc opened this issue Jun 13, 2019 · 0 comments
Assignees
@yadudoc @tskluzac
Labels
enhancement New feature or request
Milestone
0.1.0

Comments

@yadudoc
Copy link
Collaborator

yadudoc commented Jun 13, 2019

Currently the executor assumes that every worker is uniform, running the same container. However for better resource management we want the manager to be able start, replace and otherwise manage containers based on the needs of the tasks it receives.

For this to work we'll need to update several components:

  • Update task serialization logic to add a header that indicates container info
  • Managers to support workers that can disconnect
  • Manager to report online containers and resources to the interchange
  • Interchange to parse task headers and direct tasks to Managers with matching containers
  • Interchange to detect when there's insufficient active containers but enough CPU/Mem to accommodate loading more containers and send tasks to managers.
  • Interchange might need to do logic to allow sending tasks to managers so that the manager can swap containers from the cache.

There's a bunch of caching/scavenging logic that needs to happen here. Might be worthwhile to read up on distributed caching.
@yadudoc yadudoc added the enhancement New feature or request label Jun 13, 2019
@yadudoc yadudoc added this to the 0.1.0 milestone Jun 13, 2019
yadudoc added a commit that referenced this issue Aug 13, 2019
@yadudoc
Merge pull request #38 from funcx-faas/worker_update_#5
ecfda80 
Worker update #5
ZhuozhaoLi added a commit that referenced this issue Aug 14, 2019
@ZhuozhaoLi
Merge pull request #39 from funcx-faas/coupling_functions_with_contai…
27658b0 
…ners_#5

Coupling functions with containers #5
tskluzac added a commit that referenced this issue Sep 9, 2019
@tskluzac
Cleaned up variable names, removed redundant code. For PR #5.
4b1be3a
@tskluzac tskluzac mentioned this issue Sep 9, 2019
Closed
yadudoc added a commit that referenced this issue Sep 10, 2019
@yadudoc
Merge pull request #49 from funcx-faas/multi_cont_#5
49768f9 
Multi cont #5
@yadudoc yadudoc mentioned this issue Sep 10, 2019
Merged
@yadudoc yadudoc closed this as completed Sep 10, 2019
ZhuozhaoLi pushed a commit that referenced this issue Mar 23, 2020
@yadudoc
Merge pull request #38 from funcx-faas/worker_update_#5
c32a892 
Worker update #5
ZhuozhaoLi added a commit that referenced this issue Mar 23, 2020
@ZhuozhaoLi
Merge pull request #39 from funcx-faas/coupling_functions_with_contai…
ecfe856 
…ners_#5

Coupling functions with containers #5
ZhuozhaoLi pushed a commit that referenced this issue Mar 23, 2020
@yadudoc
Merge pull request #49 from funcx-faas/multi_cont_#5
876abf3 
Multi cont #5
LeiGlobus added a commit that referenced this issue Mar 30, 2023
@LeiGlobus
# This is a combination of 24 commits.
6a98cc2 
# This is the 1st commit message:

Rename dirs to compute_endpoint and compute_sdk

Update docs to reflect Globus Compute changes

Update package/deploy files for compute branding

# This is the commit message #2:

Modify imports for compute packages

# This is the commit message #3:

Rename FuncXClient to Client

# This is the commit message #4:

Rename FuncXExecutor to Executor

# This is the commit message #5:

Rename FuncxWebClient to WebClient

# This is the commit message #6:

Rename FuncXTask to ComputeTask

# This is the commit message #7:

Rename FuncXSerializer to ComputeSerializer

# This is the commit message #8:

Rename FuncxScopes to ComputeScopes

# This is the commit message #9:

Rename funcx_manager to manager

# This is the commit message #10:

Rename FuncXWorker to Worker

# This is the commit message #11:

Rename funcx_endpoint/ tests dir to endpoint/

# This is the commit message #12:

Rename FuncXFuture to ComputeFuture

# This is the commit message #13:

Update exception names for compute branding

# This is the commit message #14:

Rename get_funcx_web_client to get_web_client

# This is the commit message #15:

Fix mock paths in endpoint unit tests

# This is the commit message #16:

Rename funcx-endpoint to globus-compute-endpoint

# This is the commit message #17:

Rename funcx-worker to globus-compute-worker

# This is the commit message #18:

Rename funcx-manager to globus-compute-manager

# This is the commit message #19:

Update test config for compute branding

# This is the commit message #20:

Update misc. docstring/comments for compute branding

# This is the commit message #21:

Rename FuncxScopeBuilder to ComputeScopeBuilder

# This is the commit message #22:

Rename FuncxConsoleFormatter to ComputeConsoleFormatter

# This is the commit message #23:

Rename dirs to compute_endpoint and compute_sdk

Update docs to reflect Globus Compute changes

Rename FuncXTask to ComputeTask

Update docs to reflect Globus Compute changes

Lint fixes

Draft for funcx setup files

# This is the commit message #24:

setup and init files for funcx
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yadudoc yadudoc

@tskluzac tskluzac

Labels
enhancement New feature or request
Projects
None yet
Milestone
0.1.0
Development

No branches or pull requests
2 participants
@yadudoc @tskluzac