Big overhead on task switch with CONFIG_VIRT_CPU_ACCOUNTING_NATIVE

[chleroy]

Measurement done on a 8xx with the following app:

void *thread(void *arg)
{
	int i;

	for (i = 0; i < atoi((char*)arg); i++)
		pthread_yield();
}

int main(int argc, char **argv)
{
	pthread_t th1, th2;

	pthread_create(&th1, NULL, thread, argv[1]);
	pthread_create(&th2, NULL, thread, argv[1]);
	pthread_join(th1, NULL);
	pthread_join(th2, NULL);

	return 0;
}

Result with CONFIG_TICK_CPU_ACCOUNTING:

Performance counter stats for 'chrt -f 98 ./sched 100000' (50 runs):

      6633.062518      task-clock (msec)         #    0.955 CPUs utilized            ( +-  0.21% )
           200004      context-switches          #    0.030 M/sec                    ( +-  0.00% )

Result with CONFIG_VIRT_CPU_ACCOUNTING_NATIVE:

 Performance counter stats for 'chrt -f 98 ./sched 100000' (50 runs):

       8228.476465      task-clock (msec)         #    0.954 CPUs utilized            ( +-  0.23% )
            200004      context-switches          #    0.024 M/sec                    ( +-  0.00% )

So 1595ms additional ms are spent with CONFIG_VIRT_CPU_ACCOUNTING_NATIVE, that is a 24% overhead.

[chleroy]

A first reduction can be achieved by removing CONFIG_ARCH_HAS_SCALED_CPUTIME on targets not having SPURR and/or PURR registers.

The following result is obtained with the patch https://patchwork.ozlabs.org/patch/952605/

 Performance counter stats for 'chrt -f 98 ./sched 100000' (50 runs):

       7649.070444      task-clock (msec)         #    0.955 CPUs utilized            ( +-  0.27% )
            200004      context-switches          #    0.026 M/sec                    ( +-  0.00% )

With this change, the overhead is now only 15% over CONFIG_TICK_CPU_ACCOUNTING, so it reduces the overhead by 10 points, ie it removes 40% of the overhead.

[chleroy]

In https://patchwork.ozlabs.org/comment/1928919/ @npiggin was suggesting to restrict CONFIG_ARCH_HAS_SCALED_CPUTIME even further. Do we know exactly which configuration support that, ie have SPRN_SPURR and/or SPRN_PURR ?

[chleroy]

Commit linuxppc/linux@abcff86 merged in 4.20

The only remaining point is the comment from @npiggin mentionned above. Can anybody help with the details ?

[chleroy]

From @mpe:

PURR is Power5/6/7/8/9 and PA6T (pasemi). SPURR is Power6/7/8/9.

So we could easily flip PPC64 for PPC_BOOK3S_64, which would mean 64-bit
Book3E CPUs don't get that overhead.

Beyond that is not so simple. We probably don't need that selected for
bare metal kernels (powernv). But in practice all the distros build a
multi platform kernel with powernv+pseries anyway.

We could turn it off on G5s (PPC970), by making it depend on POWERNV ||
PSERIES || PPC_PASEMI, but I'm not sure if it's worth the trouble.

[chleroy]

Reducing scope to PPC_BOOK3S_64, see patch https://patchwork.ozlabs.org/patch/1151716/

[chleroy]

Merged in v5.4, see linuxppc/linux@b4645ff