Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce log level of k8s controller's reconciling loop's messages #771

Merged
merged 1 commit into from Nov 18, 2023
Merged

Reduce log level of k8s controller's reconciling loop's messages #771

merged 1 commit into from Nov 18, 2023

Conversation

consideRatio
Copy link
Collaborator

@consideRatio consideRatio commented Nov 17, 2023 •
edited

Current logs examplified 
[I 2023-10-25 11:13:40.135 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:13:50.178 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:13:50.178 KubeController] Cluster prod.b3a990d302d84720aae27404f6153ade scaled to 15 - creating 15 workers, deleting 0 stopped workers
[I 2023-10-25 11:13:50.328 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:14:58.490 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:14:58.491 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:14:58.870 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:14:58.870 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:14:58.879 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:14:58.879 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:14:58.880 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:14:58.880 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:14:58.888 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:14:58.888 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:14:58.889 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:14:58.889 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:14:58.892 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:14:58.892 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:14:58.894 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:14:58.895 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:14:58.907 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:14:58.907 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:15:57.600 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:15:57.601 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:15:57.657 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:15:57.657 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:15:57.671 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:15:57.672 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:15:57.747 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:15:57.747 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:15:57.748 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:15:57.748 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:15:58.001 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:15:58.002 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:15:58.497 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:15:58.497 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:16:11.540 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:16:11.541 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:16:12.843 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:16:12.844 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:16:13.215 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:16:13.215 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:16:14.000 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:16:14.000 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:16:15.771 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:16:15.772 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:16:15.903 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:16:15.903 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:16:16.059 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:16:16.059 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:16:16.362 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:16:16.362 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:19:52.978 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:19:52.978 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:26:33.145 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:26:33.145 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:33:13.468 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:33:13.468 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:35:03.417 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:35:03.418 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:35:05.897 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:35:05.897 KubeController] Cluster prod.b3a990d302d84720aae27404f6153ade scaled to 1 - deleting 0 pending and 1 stopped workers
[I 2023-10-25 11:35:05.920 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:35:07.916 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:35:07.916 KubeController] Cluster prod.b3a990d302d84720aae27404f6153ade scaled to 1 - deleting 0 pending and 13 stopped workers
[I 2023-10-25 11:35:08.026 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:35:08.899 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:35:08.899 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:38:33.606 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:38:33.606 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:39:33.708 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:39:33.708 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:43:39.620 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:43:39.621 KubeController] Shutting down prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:43:39.667 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:43:39.668 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade
[I 2023-10-25 11:43:39.668 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade
After lowering the log-level, the same logs examplified 
[I 2023-10-25 11:13:50.178 KubeController] Cluster prod.b3a990d302d84720aae27404f6153ade scaled to 15 - creating 15 workers, deleting 0 stopped workers
[I 2023-10-25 11:35:05.897 KubeController] Cluster prod.b3a990d302d84720aae27404f6153ade scaled to 1 - deleting 0 pending and 1 stopped workers
[I 2023-10-25 11:35:07.916 KubeController] Cluster prod.b3a990d302d84720aae27404f6153ade scaled to 1 - deleting 0 pending and 13 stopped workers
[I 2023-10-25 11:43:39.621 KubeController] Shutting down prod.b3a990d302d84720aae27404f6153ade

@consideRatio consideRatio merged commit 42bb373 into dask:main Nov 18, 2023
21 checks passed
@consideRatio
Copy link
Collaborator Author

Thank you @TomAugspurger for reviewing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@TomAugspurger TomAugspurger TomAugspurger approved these changes

@martindurant martindurant Awaiting requested review from martindurant

@jacobtomlinson jacobtomlinson Awaiting requested review from jacobtomlinson

Assignees
No one assigned
Labels
codebase:dask-gateway-server codebase:k8s maintenance
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Change of controller's log level of "Reconciling cluster"
2 participants
@consideRatio @TomAugspurger