[Merged by Bors] - feat: Set explicit resources on all containers #359

razvan · 2023-06-02T14:36:24Z

rust/operator/src/hdfs_controller.rs

razvan · 2023-06-11T08:46:02Z

Tests pass on kind (using the zookeeper operator 0.0.0-pr691):

--- PASS: kuttl (2682.60s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.4-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-default_listener-class-external-unstable (176.54s)
        --- PASS: kuttl/harness/cluster-operation_hadoop-latest-3.3.4-stackable0.0.0-dev_zookeeper-latest-3.8.0-stackable0.0.0-dev (232.97s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.4-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-default_listener-class-cluster-internal (168.76s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.3-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-2hdd-1ssd_listener-class-external-unstable (240.95s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.4-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-2hdd-1ssd_listener-class-external-unstable (183.74s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.4-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-2hdd-1ssd_listener-class-cluster-internal (189.96s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.4-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-default_listener-class-external-unstable (150.00s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.4-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-default_listener-class-cluster-internal (155.77s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.4-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-2hdd-1ssd_listener-class-external-unstable (156.72s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.4-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-2hdd-1ssd_listener-class-cluster-internal (158.22s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.3-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-default_listener-class-external-unstable (187.92s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.3-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-default_listener-class-cluster-internal (193.21s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.3-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-2hdd-1ssd_listener-class-external-unstable (193.00s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.3-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-2hdd-1ssd_listener-class-cluster-internal (200.23s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.3-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-default_listener-class-external-unstable (159.74s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.3-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-default_listener-class-cluster-internal (159.35s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.3-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-2hdd-1ssd_listener-class-cluster-internal (155.08s)
        --- PASS: kuttl/harness/smoke_hadoop-3.2.2-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-default_listener-class-cluster-internal (239.95s)
        --- PASS: kuttl/harness/smoke_hadoop-3.2.2-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-default_listener-class-external-unstable (169.09s)
        --- PASS: kuttl/harness/smoke_hadoop-3.2.2-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-default_listener-class-cluster-internal (136.25s)
        --- PASS: kuttl/harness/smoke_hadoop-3.2.2-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-default_listener-class-external-unstable (142.66s)
        --- PASS: kuttl/harness/smoke_hadoop-3.2.2-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-2hdd-1ssd_listener-class-cluster-internal (170.25s)
        --- PASS: kuttl/harness/orphaned-resources_hadoop-latest-3.3.4-stackable0.0.0-dev_zookeeper-latest-3.8.0-stackable0.0.0-dev (154.44s)
        --- PASS: kuttl/harness/smoke_hadoop-3.2.2-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-2hdd-1ssd_listener-class-external-unstable (185.50s)
        --- PASS: kuttl/harness/smoke_hadoop-3.2.2-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-2hdd-1ssd_listener-class-external-unstable (136.60s)
        --- PASS: kuttl/harness/logging_hadoop-3.3.3-stackable0.0.0-dev_zookeeper-latest-3.8.0-stackable0.0.0-dev (230.04s)
        --- PASS: kuttl/harness/logging_hadoop-3.3.4-stackable0.0.0-dev_zookeeper-latest-3.8.0-stackable0.0.0-dev (261.15s)
        --- PASS: kuttl/harness/logging_hadoop-3.2.2-stackable0.0.0-dev_zookeeper-latest-3.8.0-stackable0.0.0-dev (249.46s)
        --- PASS: kuttl/harness/smoke_hadoop-3.2.2-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-2hdd-1ssd_listener-class-cluster-internal (147.63s)
PASS

# Conflicts: # CHANGELOG.md

razvan

lgtm.

I think we decided that the smoke test shouldn't configure any resources explicitly and just test the defaults. I see we didn't do it for Zookeeper but only for Hive.

Pinging @sbernauer for his opinion.

maltesander · 2023-06-27T14:08:16Z

lgtm.

I think we decided that the smoke test shouldn't configure any resources explicitly and just test the defaults. I see we didn't do it for Zookeeper but only for Hive.

Pinging @sbernauer for his opinion.

Im fine with that. But there is no test that explicitly tests resources (e.g. the defaults, explicitly set) etc.

maltesander · 2023-06-27T15:16:38Z

test: https://ci.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/hdfs-operator-it-custom/88/

sbernauer

We must also change the default resources (cpu,memory,pvc) of every role

rust/operator/src/container.rs

tests/templates/kuttl/smoke/20-install-zk.yaml.j2

tests/templates/kuttl/smoke/30-assert.yaml.j2

sbernauer

Nearly done

docs/modules/hdfs/pages/usage-guide/resources.adoc

rust/operator/src/container.rs

Co-authored-by: Sebastian Bernauer <sebastian.bernauer@stackable.de>

sbernauer

Thx! LGTM when tests pass

maltesander · 2023-06-28T14:33:39Z

bors merge

Part of stackabletech/issues#394 Co-authored-by: Malte Sander <malte.sander.it@gmail.com> Co-authored-by: Sebastian Bernauer <sebastian.bernauer@stackable.de> Co-authored-by: Malte Sander <contact@maltesander.com>

bors · 2023-06-28T14:42:25Z

Pull request successfully merged into main.

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

All tests passed

Added resource quota for smoke tests.

06f83c5

razvan mentioned this pull request Jun 2, 2023

Set explicit resources on all containers stackabletech/issues#394

Closed

github-actions bot reviewed Jun 2, 2023

View reviewed changes

rust/operator/src/hdfs_controller.rs Outdated Show resolved Hide resolved

razvan added 3 commits June 6, 2023 09:48

Add res requests to utility container and update resource quotas.

e1d1620

Add res requests for side car and init containers. Smoke tests pass.

ee16eed

Add newline before EOF.

2479646

razvan marked this pull request as ready for review June 11, 2023 08:46

razvan requested a review from a team June 11, 2023 08:46

merge main

03a4d46

maltesander self-assigned this Jun 27, 2023

maltesander added 10 commits June 27, 2023 10:39

bump operator-rs

925cf28

add resources for vector container

378c9f4

regenerated charts

e4c4b5a

added resources for zkfc

bd78a02

wip for nat

2767481

bump operator-rs 0.42.2

a1d191d

init containers get resources from main app container

8b581cb

adapted tests

e8c9d88

adapted changelog

6d74d13

Merge remote-tracking branch 'origin/main' into feat/resource-quotas

cb7b64c

# Conflicts: # CHANGELOG.md

razvan commented Jun 27, 2023

View reviewed changes

add resources to create-tls-cert-bundle

cab71c0

sbernauer self-assigned this Jun 28, 2023

sbernauer changed the title ~~Smoke tests with resource quotas~~ feat: Set explicit resources on all containers Jun 28, 2023

sbernauer requested changes Jun 28, 2023

View reviewed changes

maltesander added 2 commits June 28, 2023 09:16

improve default resource limits

2009ba1

remove resources link

b9f5232

maltesander and others added 9 commits June 28, 2023 09:38

attempt to fix failing test asserts

23c5908

remove zookeeper resources to test defaults

539f10f

change side car container order

ca3ab4a

wip test - only run smoke tests in cloud

a2951b4

enable all tests

ebb5ef3

unleash the chaosmonkey later!

7689ce0

add comments to resource usage

36e7f33

Make default resources depend on role

52b504b

fix storage test

650e769

sbernauer requested changes Jun 28, 2023

View reviewed changes

docs/modules/hdfs/pages/usage-guide/resources.adoc Show resolved Hide resolved

rust/operator/src/container.rs Outdated Show resolved Hide resolved

maltesander and others added 3 commits June 28, 2023 16:25

Update rust/operator/src/container.rs

7865634

Co-authored-by: Sebastian Bernauer <sebastian.bernauer@stackable.de>

fix code

0c14c80

add memory to resource example

a0c9448

maltesander requested a review from sbernauer June 28, 2023 14:30

sbernauer approved these changes Jun 28, 2023

View reviewed changes

bors bot changed the title ~~feat: Set explicit resources on all containers~~ [Merged by Bors] - feat: Set explicit resources on all containers Jun 28, 2023

bors bot closed this Jun 28, 2023

bors bot deleted the feat/resource-quotas branch June 28, 2023 14:42

lfrancke added the release/2023-07 label Jul 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Merged by Bors] - feat: Set explicit resources on all containers #359

[Merged by Bors] - feat: Set explicit resources on all containers #359

razvan commented Jun 2, 2023 •

edited

Loading

razvan commented Jun 11, 2023

razvan left a comment

maltesander commented Jun 27, 2023

maltesander commented Jun 27, 2023 •

edited

Loading

sbernauer left a comment

sbernauer left a comment

sbernauer left a comment

maltesander commented Jun 28, 2023

bors bot commented Jun 28, 2023

[Merged by Bors] - feat: Set explicit resources on all containers #359

[Merged by Bors] - feat: Set explicit resources on all containers #359

Conversation

razvan commented Jun 2, 2023 • edited Loading

razvan commented Jun 11, 2023

razvan left a comment

Choose a reason for hiding this comment

maltesander commented Jun 27, 2023

maltesander commented Jun 27, 2023 • edited Loading

sbernauer left a comment

Choose a reason for hiding this comment

sbernauer left a comment

Choose a reason for hiding this comment

sbernauer left a comment

Choose a reason for hiding this comment

maltesander commented Jun 28, 2023

bors bot commented Jun 28, 2023

razvan commented Jun 2, 2023 •

edited

Loading

maltesander commented Jun 27, 2023 •

edited

Loading