CPU is busy even when idle

[mazar]

Description

Running 0.14.6 for a few weeks with no issues, I decided to upgrade to 0.14.12. All looks good except CPU is busy even when there is no flow running. This was not the case with 0.14.6. Here is the output from docker stats --format "table {{.Name}}\t{{.CPUPerc}}"

NAME                CPU %
tmp_ui_1            0.00%
tmp_towel_1         0.00%
tmp_apollo_1        13.08%
tmp_graphql_1       3.56%  
tmp_hasura_1        11.23%
tmp_postgres_1      7.40%

And here is the docker version:

docker version
Client:
 Version:           19.03.6
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        369ce74a3c
 Built:             Fri Dec 18 12:21:44 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.6
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       369ce74a3c
  Built:            Thu Dec 10 13:23:49 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu1~18.04.4
  GitCommit:        
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:        
 docker-init:
  Version:          0.18.0
  GitCommit:        

And docker-compose version:

docker-compose version 1.28.5, build c4eb3a1f

Then I upgraded my docker to latest version:

Client: Docker Engine - Community
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        55c4c88
 Built:             Tue Mar  2 20:18:05 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Tue Mar  2 20:16:00 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

But behavior did not change.

I could not downgrade to previous working version (0.14.6) due to DB migration not allowing downgrade. So I tried on a different machine (Mac laptop) with same docker version (20.10.5).
I upgraded my local Prefect from 0.14.6 to 0.14.12, and high CPU could be observed:

NAME           CPU %
t_ui_1         0.01%
t_towel_1      0.01%
t_apollo_1     3.96%
t_graphql_1    4.62%
t_hasura_1     3.36%
t_postgres_1   7.71%

And then downgraded to 0.14.6 and CPU looks normal:

NAME           CPU %
t_ui_1         0.02%
t_apollo_1     0.00%
t_towel_1      0.00%
t_graphql_1    0.48%
t_hasura_1     0.80%
t_postgres_1   0.53%

Expected Behavior

When Prefect is idle (no flows are running), CPU should not be busy, as shown above.

Reproduction

1- Install Prefect 0.14.6 (and maybe other versions?)
2- Use docker stats to observe CPU usage
3- Install Prefect 0.14.12
4- Use docker stats to observe CPU usage

Environment

Problem observed on:

Ubuntu:

Linux ip-172-31-49-216 5.4.0-1039-aws #41~18.04.1-Ubuntu SMP Fri Feb 26 11:20:14 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Python 3.7.6

And

macos:
Darwin laptop 20.3.0 Darwin Kernel Version 20.3.0: Thu Jan 21 00:07:06 PST 2021; root:xnu-7195.81.3~1/RELEASE_X86_64 x86_64

Python 3.8.5

[cicdw]

Hi @mazar - thanks for the detailed issue! These numbers are not concerning or abnormally high, and are caused by the new health checks for the various services that were introduced in 0.14.7.

[mazar]

@cicdw I respectfully disagree! The numbers are concerning for a machine with 4 cores, with absolutely no load, even no agents are connected to it.

[mazar]

This issue was fixed in 0.14.16 in following commits:
PrefectHQ/prefect@dcc7804
PrefectHQ/prefect@c825086

[zanieb]

Hey @mazar -- thank you for following up! I agree that those numbers were not ideal but forgot to reply earlier. I knew I'd bump them higher when I had the time 😄 Glad to hear those commits fixed things up.