[New Scheduler] Manage container creation #5074

KeonHee · 2021-03-02T16:09:35Z

Description

This module determines which invoker to create the container from.

My changes affect the following components

Types of changes

Bug fix (generally a non-breaking change which closes an issue).
Enhancement or new feature (adds new functionality).
Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

I signed an Apache CLA.
I reviewed the style guides and followed the recommendations (Travis CI will check :).
I added tests to cover my changes.
My changes require further changes to the documentation.
I updated the documentation where necessary.

codecov-io · 2021-03-02T17:02:33Z

Codecov Report

Merging #5074 (9f4708d) into master (cd6fded) will decrease coverage by 6.07%.
The diff coverage is 74.38%.

@@            Coverage Diff             @@
##           master    #5074      +/-   ##
==========================================
- Coverage   81.25%   75.17%   -6.08%     
==========================================
  Files         218      222       +4     
  Lines       10700    11010     +310     
  Branches      450      484      +34     
==========================================
- Hits         8694     8277     -417     
- Misses       2006     2733     +727

Impacted Files	Coverage Δ
.../apache/openwhisk/core/controller/Controller.scala	`54.80% <ø> (ø)`
...enwhisk/core/loadBalancer/InvokerSupervision.scala	`88.81% <ø> (-0.46%)`	⬇️
...che/openwhisk/core/loadBalancer/LoadBalancer.scala	`90.00% <ø> (+13.07%)`	⬆️
...e/loadBalancer/ShardingContainerPoolBalancer.scala	`80.79% <ø> (ø)`
...k/core/containerpool/v2/InvokerHealthManager.scala	`75.00% <ø> (-0.38%)`	⬇️
.../org/apache/openwhisk/core/connector/Message.scala	`64.76% <23.63%> (-14.22%)`	⬇️
.../main/scala/org/apache/openwhisk/core/WarmUp.scala	`64.28% <64.28%> (ø)`
...la/org/apache/openwhisk/common/TransactionId.scala	`94.25% <75.00%> (+0.13%)`	⬆️
...in/scala/org/apache/openwhisk/common/Message.scala	`77.77% <77.77%> (ø)`
...sk/core/scheduler/container/ContainerManager.scala	`86.76% <86.76%> (ø)`
... and 36 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cd6fded...9f4708d. Read the comment docs.

KeonHee · 2021-04-06T18:33:17Z

The standalone test continues to fail because an error occurred while pulling openwhisk/action-nodejs-v10:nightly.

$ docker pull openwhisk/action-nodejs-v10:nightly
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

style95 · 2021-04-07T00:55:49Z

The standalone test continues to fail because an error occurred while pulling openwhisk/action-nodejs-v10:nightly.
$ docker pull openwhisk/action-nodejs-v10:nightly
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

I worked around this by pulling such kind images in advance before starting tests.

ningyougang · 2021-04-14T06:06:20Z

common/scala/src/main/scala/org/apache/openwhisk/core/connector/Message.scala

+    result = prime * result + tags.hashCode()
+    result = prime * result + dedicatedNamespaces.hashCode()
+    result
+  }


Regarding InvokerResourceMessage's equals and hashCode, it seems we have no need to override these 2 methods.

ningyougang · 2021-04-28T02:48:02Z

ansible/group_vars/all

@@ -139,6 +139,7 @@ registry:
  confdir: "{{ config_root_dir }}/registry"

 kafka:
+  topicsPrefix: "{{ kafka_topics_prefix | default('') }}"


Add prefix for kafka topic feature is already implemented in this pr: #5062

style95

LGTM
@KeonHee

Can we add some design documents for this component especially regarding the container creation/deletion process and warmup process here?
https://cwiki.apache.org/confluence/display/OPENWHISK/Component+Design

KeonHee · 2021-04-29T06:23:02Z

@style95 Sure. I will upload it ASAP

KeonHee · 2021-05-06T07:45:38Z

I wrote a document: https://cwiki.apache.org/confluence/display/OPENWHISK/ContainerManager

style95 · 2021-05-06T07:57:23Z

I wrote a document: https://cwiki.apache.org/confluence/display/OPENWHISK/ContainerManager

Can we add a section about warmup?

codecov-commenter · 2021-05-07T02:42:59Z

Codecov Report

Merging #5074 (d4a43d6) into master (9c445f3) will decrease coverage by 5.56%.
The diff coverage is 84.49%.

@@            Coverage Diff             @@
##           master    #5074      +/-   ##
==========================================
- Coverage   81.11%   75.54%   -5.57%     
==========================================
  Files         214      225      +11     
  Lines       10913    11423     +510     
  Branches      469      501      +32     
==========================================
- Hits         8852     8630     -222     
- Misses       2061     2793     +732

Impacted Files	Coverage Δ
.../apache/openwhisk/core/controller/Controller.scala	`55.66% <ø> (ø)`
...enwhisk/core/loadBalancer/InvokerSupervision.scala	`88.81% <ø> (-0.46%)`	⬇️
...che/openwhisk/core/loadBalancer/LoadBalancer.scala	`90.00% <ø> (+13.07%)`	⬆️
...e/loadBalancer/ShardingContainerPoolBalancer.scala	`80.79% <ø> (ø)`
...k/core/containerpool/v2/InvokerHealthManager.scala	`75.00% <ø> (-0.38%)`	⬇️
.../main/scala/org/apache/openwhisk/core/WarmUp.scala	`64.28% <64.28%> (ø)`
...la/org/apache/openwhisk/common/TransactionId.scala	`94.25% <75.00%> (+0.13%)`	⬆️
...in/scala/org/apache/openwhisk/common/Message.scala	`77.77% <77.77%> (ø)`
...sk/core/scheduler/container/ContainerManager.scala	`86.76% <86.76%> (ø)`
...in/scala/org/apache/openwhisk/common/Logging.scala	`77.22% <100.00%> (-8.35%)`	⬇️
... and 42 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9c445f3...d4a43d6. Read the comment docs.

KeonHee added the scheduler label Mar 2, 2021

KeonHee changed the title ~~[WIP] [New Scheduler] Manage container creation~~ [New Scheduler] Manage container creation Mar 22, 2021

KeonHee changed the title ~~[New Scheduler] Manage container creation~~ [WIP] [New Scheduler] Manage container creation Mar 23, 2021

KeonHee changed the title ~~[WIP] [New Scheduler] Manage container creation~~ [New Scheduler] Manage container creation Mar 23, 2021

ningyougang reviewed Apr 14, 2021

View reviewed changes

ningyougang reviewed Apr 28, 2021

View reviewed changes

style95 approved these changes Apr 29, 2021

View reviewed changes

KeonHee added 12 commits May 25, 2021 15:49

Manage container creation

adbe449

Add ContainerManager Test

ac9938d

Add dedicated namespace

82a9437

Remove namespace

020e6fa

Apply scala fmt

ad6c355

Add dedicatedNamespaces filter

0de9b51

Add dedicatedNamespaces test

13a35a2

Move InvokerState to common

c194857

Unify InvokerHealth message

cf1555f

Add configuration for test

5c965d5

Add license header

35262d4

Remove compare InvokerResourceMessage

d4a43d6

style95 merged commit f1829e1 into apache:master May 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[New Scheduler] Manage container creation #5074

[New Scheduler] Manage container creation #5074

KeonHee commented Mar 2, 2021 •

edited

codecov-io commented Mar 2, 2021 •

edited

KeonHee commented Apr 6, 2021

style95 commented Apr 7, 2021

ningyougang Apr 14, 2021

ningyougang Apr 28, 2021 •

edited

style95 left a comment

KeonHee commented Apr 29, 2021

KeonHee commented May 6, 2021

style95 commented May 6, 2021

codecov-commenter commented May 7, 2021 •

edited

[New Scheduler] Manage container creation #5074

[New Scheduler] Manage container creation #5074

Conversation

KeonHee commented Mar 2, 2021 • edited

Description

My changes affect the following components

Types of changes

Checklist:

codecov-io commented Mar 2, 2021 • edited

Codecov Report

KeonHee commented Apr 6, 2021

style95 commented Apr 7, 2021

ningyougang Apr 14, 2021

Choose a reason for hiding this comment

ningyougang Apr 28, 2021 • edited

Choose a reason for hiding this comment

style95 left a comment

Choose a reason for hiding this comment

KeonHee commented Apr 29, 2021

KeonHee commented May 6, 2021

style95 commented May 6, 2021

codecov-commenter commented May 7, 2021 • edited

Codecov Report

KeonHee commented Mar 2, 2021 •

edited

codecov-io commented Mar 2, 2021 •

edited

ningyougang Apr 28, 2021 •

edited

codecov-commenter commented May 7, 2021 •

edited