Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable prometheus metrics for katib-controller #717

Merged
merged 3 commits into from Aug 16, 2019

Conversation

@hougangliu
Copy link
Member

commented Aug 13, 2019

Upgrade controller-runtime to 0.1.9 and enable prometheus metrics for katib-controller

# curl http://10.0.149.224:8080/metrics
...
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 1.8476192e+07
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.819165e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.671168e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.671168e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.3072888e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 21
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 5.09
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 12
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 4.6043136e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.56568718538e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.41164544e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1
...

This change is Reviewable

@johnugeorge

This comment has been minimized.

Copy link
Member

commented Aug 13, 2019

Don't we need katib specific metrics to be exposed?

@hougangliu hougangliu referenced this pull request Aug 13, 2019
3 of 3 tasks complete
@hougangliu

This comment has been minimized.

Copy link
Member Author

commented Aug 13, 2019

Don't we need katib specific metrics to be exposed?

This PR only exposes controller-runtime default metrics for controller. I will submit other PRs for experiments/trials etc.

@hougangliu

This comment has been minimized.

Copy link
Member Author

commented Aug 13, 2019

/test kubeflow-katib-presubmit

1 similar comment
@hougangliu

This comment has been minimized.

Copy link
Member Author

commented Aug 14, 2019

/test kubeflow-katib-presubmit

@hougangliu

This comment has been minimized.

Copy link
Member Author

commented Aug 14, 2019

Copy link
Member

left a comment

/lgtm
/retest

@k8s-ci-robot k8s-ci-robot added the lgtm label Aug 14, 2019
@johnugeorge

This comment has been minimized.

Copy link
Member

commented Aug 14, 2019

/lgtm

@johnugeorge

This comment has been minimized.

Copy link
Member

commented Aug 14, 2019

/approve

@johnugeorge

This comment has been minimized.

Copy link
Member

commented Aug 14, 2019

/retest

1 similar comment
@johnugeorge

This comment has been minimized.

Copy link
Member

commented Aug 14, 2019

/retest

@hougangliu

This comment has been minimized.

Copy link
Member Author

commented Aug 14, 2019

/test kubeflow-katib-presubmit

@johnugeorge

This comment has been minimized.

Copy link
Member

commented Aug 14, 2019

/retest

@hougangliu

This comment has been minimized.

Copy link
Member Author

commented Aug 14, 2019

/test kubeflow-katib-presubmit

@johnugeorge

This comment has been minimized.

Copy link
Member

commented Aug 14, 2019

/retest

@hougangliu

This comment has been minimized.

Copy link
Member Author

commented Aug 14, 2019

/test kubeflow-katib-presubmit

2 similar comments
@hougangliu

This comment has been minimized.

Copy link
Member Author

commented Aug 15, 2019

/test kubeflow-katib-presubmit

@gaocegege

This comment has been minimized.

Copy link
Member

commented Aug 15, 2019

/test kubeflow-katib-presubmit

@k8s-ci-robot k8s-ci-robot removed the lgtm label Aug 15, 2019
@hougangliu

This comment has been minimized.

Copy link
Member Author

commented Aug 15, 2019

Disable UI build so that the build error will not block katib CI.
@andreyvelich will be back to fix #699

@hougangliu

This comment has been minimized.

Copy link
Member Author

commented Aug 15, 2019

/test kubeflow-katib-presubmit

@gaocegege

This comment has been minimized.

Copy link
Member

commented Aug 15, 2019

/retest

2 similar comments
@hougangliu

This comment has been minimized.

Copy link
Member Author

commented Aug 15, 2019

/retest

@hougangliu

This comment has been minimized.

Copy link
Member Author

commented Aug 15, 2019

/retest

@hougangliu

This comment has been minimized.

Copy link
Member Author

commented Aug 16, 2019

/retest

@johnugeorge

This comment has been minimized.

Copy link
Member

commented Aug 16, 2019

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm label Aug 16, 2019
@johnugeorge

This comment has been minimized.

Copy link
Member

commented Aug 16, 2019

/approve

@k8s-ci-robot

This comment has been minimized.

Copy link

commented Aug 16, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johnugeorge

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 45124c1 into kubeflow:master Aug 16, 2019
5 of 6 checks passed
5 of 6 checks passed
tide Not mergeable.
Details
Travis CI - Pull Request Build Passed
Details
cla/google All necessary CLAs are signed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
coverage/coveralls Coverage increased (+0.1%) to 58.193%
Details
kubeflow-katib-presubmit Job succeeded.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.