[WIP][don't merge] Dev:1st impl for integration test #275

SamYuan1990 · 2022-10-03T13:12:38Z

Details for this PR
e2e folder with end to end testing, will run with two models, base on environment kepler_address.

build kepler during test process and run it targets on local.
run test case targets on cluster after cluster port forwarding.

Signed-off-by: Sam Yuan yy19902439@126.com

SamYuan1990 · 2022-10-03T13:37:37Z

@marceloamaral , if I am correct, in integration test, we will deploy kepler with port 9102 right? could you please help with error https://github.com/sustainable-computing-io/kepler/actions/runs/3174554723/jobs/5171489975#step:8:28

cmd/exporter.go

SamYuan1990 · 2022-10-03T13:38:41Z

Sorry, for asking for help on this WIP pr.

cmd/exporter.go

SamYuan1990 · 2022-10-04T06:48:18Z

pkg/collector/reader.go

@@ -104,8 +104,12 @@ func (c *Collector) resetCurrValue() {

 // resetBPFTables reset BPF module's tables
 func (c *Collector) resetBPFTables() {
-	c.modules.Table.DeleteAll()
-	c.modules.TimeTable.DeleteAll()
+	if c.modules != nil && c.modules.Table != nil {


@rootfs , do you know where we init modules and tables?

the tables are auto filled; the delete functions remove existing table entries, the ebpf program will fill the tables again.

SamYuan1990 · 2022-10-04T06:49:06Z

pkg/power/acpi/util_linux.go

@@ -0,0 +1,94 @@
+//go:build linux
+// +build linux


@rootfs ,those path are linux specific, so we have to split at here for different device/operator system?

@marceloamaral any idea? maybe a stub here too?

SamYuan1990 · 2022-10-04T06:49:49Z

pkg/attacher/bcc_attacher_stub.go

@@ -66,7 +62,7 @@ var (
 )

 func AttachBPFAssets() (*BpfModuleTables, error) {
-	return nil, fmt.Errorf("no bcc build tag")
+	return nil, nil


@rootfs , here is the pre defined error for without bcc tag.

I see. This could make a case for a new bcc stub that emit mock data

SamYuan1990 · 2022-10-04T07:34:20Z

@rootfs , @marceloamaral , it seems we got error at https://github.com/sustainable-computing-io/kepler/actions/runs/3180212481/jobs/5183521776#step:7:17 could you please advise?

the network port forwards for pod failed

SamYuan1990 · 2022-10-04T07:58:47Z

topic may related https://github.com/kubernetes-sigs/kind/search?q=failed+to+execute+portforward+in+network+namespace+%22host%22&type=issues

rootfs · 2022-10-05T13:21:49Z

topic may related https://github.com/kubernetes-sigs/kind/search?q=failed+to+execute+portforward+in+network+namespace+%22host%22&type=issues

Could ClusterIP: None be causing trouble?

SamYuan1990 · 2022-10-05T14:00:23Z

topic may related https://github.com/kubernetes-sigs/kind/search?q=failed+to+execute+portforward+in+network+namespace+%22host%22&type=issues

Could ClusterIP: None be causing trouble?

I give a try by a new commit, but seems not the root cause.
from the log, I suppose there some error happened during ip v4 ip v6 transform.

SamYuan1990 · 2022-10-05T14:53:43Z

@rootfs , I am now try to force KIND use IPv4 to avoid:
https://kind.sigs.k8s.io/docs/user/known-issues/#ipv6-port-forwarding

Signed-off-by: Sam Yuan <yy19902439@126.com>

rootfs · 2022-10-06T17:45:35Z

@BenTheElder Thank you Ben for the help! The application should be running and listening to the port (in this Service).

BenTheElder · 2022-10-06T18:15:59Z

port-forward forwards to pods (not services), it will attempt to open a connection from within the pod's namespace, the tunnel looks roughly like:

kubectl <=> API server <=> kubelet <=> CRI <=> socket opened within target pod. IIRC kubernetes will always do this on localhost. The specific implementation in this case will be containerd's.

My KIND co-maintainer aojea previously patched this, because it used to literally use socat and supported IPv4 only. The current implementation has "happy eyeballs" IPv4+IPv6 (because Go) IIRC.

I'm noticing that the pod seems to be configured in hostNetwork and looks like the process should be bound to 0.0.0.0 ...

I'm not sure if we / Kubernetes have tests for hostNetwork port forward, it's possible that's broken in KIND (or containerd), otherwise unless the app isn't listening yet or the wrong pod is selected, I'm at a loss ...

rootfs · 2022-10-06T19:02:11Z

Good info @BenTheElder! So based on your feedback, a more appropriate CI deployment is to use pod network. @SamYuan1990 can you give that a try? Thanks!

SamYuan1990 · 2022-10-07T01:52:51Z

well, I also found some thing on github runner.
if I am right, github agent doesn't support ipv6 for now.
actions/runner-images#668

also kubernetes/kubectl#1169 from Jan reported at kubectl

as a short summary, we even make the case more complex as we use docker and ref to kubernetes-sigs/kind#2953 (comment)

Kubernetes within kind is using containerd.

ref:

and I suppose we need to make our life easier, by reduce components on this long tunnel.
By put test application into a pod, and use k8s network. As everything happens in side kind/k8s, we don't need to worry about the long tunnel across different application and owners.

BenTheElder · 2022-10-07T02:01:44Z

Instead of kubectl port forward, you could try configuring a port forward in kind and use a matching nodePort, similar to https://kind.sigs.k8s.io/docs/user/ingress/

kubectl port forward is more of a debugging tool and is not the most robust, though i'd still expect it to work here 🤔 , but as a general rule I wouldn't use it for anything load bearing with any cluster, even if you use the API directly.

Something like:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- extraPortMappings:
  - containerPort: 30001
    hostPort: 30001
    protocol: TCP

Or actually since the pod appears(?) to be in hostNetwork and you have a single node:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- extraPortMappings:
  - containerPort: 9102
    hostPort: 9102

I.E. adds -p 127.0.0.1:9102:9102/TCP to running the kind node.

if I am right, github agent doesn't support ipv6 for now.
actions/runner-images#668

We run some of our CI in GitHub actions and lack of IPv6 is not an issue, the ipv6 port forward connect is a red-herring, it's portforward is trying both ipv4 and ipv6 and both are failing. You can use kind / kubernetes without ipv6, if anything optional ipv6 in KIND involved a lot of fixing tests and Kubernetes, and then again in conjunction with the folks working on dualstack.

Signed-off-by: Sam Yuan <yy19902439@126.com>

SamYuan1990 · 2022-10-07T02:17:48Z

Instead of kubectl port forward, you could try configuring a port forward in kind and use a matching nodePort, similar to https://kind.sigs.k8s.io/docs/user/ingress/

kubectl port forward is more of a debugging tool and is not the most robust, though i'd still expect it to work here 🤔 , but as a general rule I wouldn't use it for anything load bearing with any cluster, even if you use the API directly.

Something like:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- extraPortMappings:
  - containerPort: 30001
    hostPort: 30001
    protocol: TCP
Or actually since the pod appears(?) to be in hostNetwork and you have a single node:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- extraPortMappings:
  - containerPort: 9102
    hostPort: 9102
I.E. adds -p 127.0.0.1:9102:9102/TCP to running the kind node.

if I am right, github agent doesn't support ipv6 for now.
actions/runner-images#668

We run some of our CI in GitHub actions and lack of IPv6 is not an issue, the ipv6 port forward connect is a red-herring, it's portforward is trying both ipv4 and ipv6 and both are failing. You can use kind / kubernetes without ipv6, if anything optional ipv6 in KIND involved a lot of fixing tests and Kubernetes, and then again in conjunction with the folks working on dualstack.

let me try

Signed-off-by: Sam Yuan <yy19902439@126.com>

SamYuan1990 · 2022-10-07T03:01:27Z

@rootfs , @marceloamaral , take a look at https://github.com/sustainable-computing-io/kepler/actions/runs/3201921603/jobs/5230372207#step:8:13 pls

Signed-off-by: Sam Yuan <yy19902439@126.com>

SamYuan1990 · 2022-10-07T04:48:50Z

some updates:

tried with https://kind.sigs.k8s.io/docs/user/configuration/#extra-port-mappings but, The Service "kepler-exporter" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when type is 'ClusterIP' should we use ClusterIP together with extra-port-mappings?
fixed one panic error, but why pod in panic passed kubectl rollout status daemonset kepler-exporter -n kepler --timeout 180s ? question here as the panic will make the service down, which breaks livenessProbe.
once panic been fixed, the port forward works. @_@

BenTheElder · 2022-10-07T06:07:46Z

The Service "kepler-exporter" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when type is 'ClusterIP' should we use ClusterIP together with extra-port-mappings?

nodePort can only be set on a service of type NodePort. shouldn't be related to the node level port mappings.

I think you can skip the service though and just map 9102 from the host to the kind node since the pod appears to be in hostNetwork.

marceloamaral · 2022-10-07T06:14:39Z

I've been using port-forward with no problem on a kind cluster.
So probably the problem here was just related to the panic error.

The command kubectl rollout status daemonset kepler-exporter -n kepler --timeout 180s verifies if the container has been started, but will not check if the container failed shortly after starting....

Maybe we should sleep and then check for failed containers....

rootfs · 2022-10-07T13:18:59Z

@SamYuan1990 now just a couple of lint error, ready to merge

SamYuan1990 · 2022-10-07T13:20:47Z

@SamYuan1990 now just a couple of lint error, ready to merge

no no no.... I am going to bed... and don't merge it for now. please take a look at #283 1st.

SamYuan1990 · 2022-10-11T16:24:00Z

close and move to #292 without issue. thanks guys.

SamYuan1990 changed the title ~~Dev:1st impl for integration test~~ [WIP] Dev:1st impl for integration test Oct 3, 2022

SamYuan1990 commented Oct 3, 2022

View reviewed changes

cmd/exporter.go Outdated Show resolved Hide resolved

SamYuan1990 requested review from marceloamaral and rootfs October 3, 2022 13:38

rootfs reviewed Oct 3, 2022

View reviewed changes

cmd/exporter.go Show resolved Hide resolved

SamYuan1990 force-pushed the integrationTest1stImpl branch from 4b22ac7 to 0dcd6a1 Compare October 4, 2022 06:39

SamYuan1990 commented Oct 4, 2022

View reviewed changes

SamYuan1990 force-pushed the integrationTest1stImpl branch 2 times, most recently from 0e841c2 to 3fe3c52 Compare October 4, 2022 07:27

SamYuan1990 mentioned this pull request Oct 4, 2022

fails to use port-forward with kind in githubaction kubernetes-sigs/kind#2953

Closed

SamYuan1990 added 3 commits October 6, 2022 09:54

Dev:1st impl for integration test

c8d6231

Signed-off-by: Sam Yuan <yy19902439@126.com>

try with InitConfiguration

40dda33

Signed-off-by: Sam Yuan <yy19902439@126.com>

try kubeProxyMode

40babef

Signed-off-by: Sam Yuan <yy19902439@126.com>

SamYuan1990 force-pushed the integrationTest1stImpl branch from 28d17b4 to 40babef Compare October 6, 2022 01:54

SamYuan1990 added 7 commits October 6, 2022 10:15

fix up

aefb527

Signed-off-by: Sam Yuan <yy19902439@126.com>

fix up

ae1550f

Signed-off-by: Sam Yuan <yy19902439@126.com>

fix up

63846a1

Signed-off-by: Sam Yuan <yy19902439@126.com>

fix up

d898516

Signed-off-by: Sam Yuan <yy19902439@126.com>

fix up

f73483e

Signed-off-by: Sam Yuan <yy19902439@126.com>

fix up

4c4ee06

Signed-off-by: Sam Yuan <yy19902439@126.com>

try calico

b8f8d50

Signed-off-by: Sam Yuan <yy19902439@126.com>

try with extraPortMappings

f489bff

Signed-off-by: Sam Yuan <yy19902439@126.com>

SamYuan1990 added 3 commits October 7, 2022 10:36

just try out

7d5c628

Signed-off-by: Sam Yuan <yy19902439@126.com>

change timeout to 180s

8fa3e47

Signed-off-by: Sam Yuan <yy19902439@126.com>

fix up

43d20da

Signed-off-by: Sam Yuan <yy19902439@126.com>

SamYuan1990 added 2 commits October 7, 2022 11:03

fix up by unmount volume for test

33bf5bf

Signed-off-by: Sam Yuan <yy19902439@126.com>

fix up

bea64f6

Signed-off-by: Sam Yuan <yy19902439@126.com>

SamYuan1990 mentioned this pull request Oct 7, 2022

make bcc logic part more robust #282

Closed

SamYuan1990 added 3 commits October 7, 2022 11:45

fix up

56a0fae

Signed-off-by: Sam Yuan <yy19902439@126.com>

fix up remove panic invoke

fc4a7d7

Signed-off-by: Sam Yuan <yy19902439@126.com>

fix up

322cf93

Signed-off-by: Sam Yuan <yy19902439@126.com>

marceloamaral mentioned this pull request Oct 7, 2022

Add panic handling when attaching the bpf program to do not termine kepler #283

Merged

SamYuan1990 added 3 commits October 7, 2022 12:19

fix up

9038f16

Signed-off-by: Sam Yuan <yy19902439@126.com>

fix up

713ab94

Signed-off-by: Sam Yuan <yy19902439@126.com>

fix up

26ad247

Signed-off-by: Sam Yuan <yy19902439@126.com>

SamYuan1990 changed the title ~~[WIP] Dev:1st impl for integration test~~ [WIP][don't merge] Dev:1st impl for integration test Oct 7, 2022

SamYuan1990 mentioned this pull request Oct 11, 2022

impl for integration test #292

Merged

SamYuan1990 closed this Oct 11, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP][don't merge] Dev:1st impl for integration test #275

[WIP][don't merge] Dev:1st impl for integration test #275

SamYuan1990 commented Oct 3, 2022

SamYuan1990 commented Oct 3, 2022

SamYuan1990 commented Oct 3, 2022

SamYuan1990 Oct 4, 2022

rootfs Oct 6, 2022

SamYuan1990 Oct 4, 2022 •

edited

rootfs Oct 6, 2022

SamYuan1990 Oct 4, 2022

rootfs Oct 6, 2022

SamYuan1990 commented Oct 4, 2022 •

edited

SamYuan1990 commented Oct 4, 2022

rootfs commented Oct 5, 2022

SamYuan1990 commented Oct 5, 2022

SamYuan1990 commented Oct 5, 2022

rootfs commented Oct 6, 2022

BenTheElder commented Oct 6, 2022

rootfs commented Oct 6, 2022

SamYuan1990 commented Oct 7, 2022 •

edited

BenTheElder commented Oct 7, 2022 •

edited

SamYuan1990 commented Oct 7, 2022

SamYuan1990 commented Oct 7, 2022

SamYuan1990 commented Oct 7, 2022

BenTheElder commented Oct 7, 2022

marceloamaral commented Oct 7, 2022

rootfs commented Oct 7, 2022

SamYuan1990 commented Oct 7, 2022

SamYuan1990 commented Oct 11, 2022

[WIP][don't merge] Dev:1st impl for integration test #275

[WIP][don't merge] Dev:1st impl for integration test #275

Conversation

SamYuan1990 commented Oct 3, 2022

SamYuan1990 commented Oct 3, 2022

SamYuan1990 commented Oct 3, 2022

SamYuan1990 Oct 4, 2022

Choose a reason for hiding this comment

rootfs Oct 6, 2022

Choose a reason for hiding this comment

SamYuan1990 Oct 4, 2022 • edited

Choose a reason for hiding this comment

rootfs Oct 6, 2022

Choose a reason for hiding this comment

SamYuan1990 Oct 4, 2022

Choose a reason for hiding this comment

rootfs Oct 6, 2022

Choose a reason for hiding this comment

SamYuan1990 commented Oct 4, 2022 • edited

SamYuan1990 commented Oct 4, 2022

rootfs commented Oct 5, 2022

SamYuan1990 commented Oct 5, 2022

SamYuan1990 commented Oct 5, 2022

rootfs commented Oct 6, 2022

BenTheElder commented Oct 6, 2022

rootfs commented Oct 6, 2022

SamYuan1990 commented Oct 7, 2022 • edited

BenTheElder commented Oct 7, 2022 • edited

SamYuan1990 commented Oct 7, 2022

SamYuan1990 commented Oct 7, 2022

SamYuan1990 commented Oct 7, 2022

BenTheElder commented Oct 7, 2022

marceloamaral commented Oct 7, 2022

rootfs commented Oct 7, 2022

SamYuan1990 commented Oct 7, 2022

SamYuan1990 commented Oct 11, 2022

SamYuan1990 Oct 4, 2022 •

edited

SamYuan1990 commented Oct 4, 2022 •

edited

SamYuan1990 commented Oct 7, 2022 •

edited

BenTheElder commented Oct 7, 2022 •

edited