[WIP] buildkit integration

[ktock]

This is an experimental integration with buildkit for speeding up fetching base image.

Our patched version of buildkit is here.

This commit includes benchmark scripts to measure the time for buliding sample images.
See this doc for manual testing.

Though it seems good for the lazy distribution of the base images, we currently have two problems for exporting output image.

1. Archiving layers takes a long time. It is because of the low READ performance including fetching contents from the registry.
2. We only support exporting type tar. It is because when we use remote snapshotters, containerd doesn't store the image contents in the content store but buildkit needs to use these contents for exporting images.

[github-actions]

Benchmarking Result

Runs on the ubuntu-18.04 runner on Github Actions.

python:3.7

mode	pull(sec)	create(sec)	run(sec)
legacy	30.80867099761963	0.02688884735107422	0.1361839771270752
stargz	2.2660980224609375	0.05379605293273926	1.0895819664001465
estargz	2.6346559524536133	0.31299591064453125	0.17602109909057617

gcc:9.2.0

mode	pull(sec)	create(sec)	run(sec)
legacy	33.24489402770996	0.032514095306396484	0.14731407165527344
stargz	2.3038840293884277	0.05231308937072754	2.3267719745635986
estargz	2.2181098461151123	0.03937196731567383	2.003551959991455

rethinkdb:2.3.6

mode	pull(sec)	create(sec)	run(sec)
legacy	8.111081838607788	0.02593088150024414	0.12709808349609375
stargz	1.1628179550170898	0.04550909996032715	1.7875571250915527
estargz	1.4818990230560303	1.3497328758239746	0.17242002487182617

glassfish:4.1-jdk8

mode	pull(sec)	create(sec)	run(sec)
legacy	27.240201950073242	0.02878880500793457	2.4394888877868652
stargz	2.4255881309509277	0.05092787742614746	16.81278896331787
estargz	2.556001901626587	7.158114194869995	2.92494797706604

[github-actions]

Buildkit integration performance(without exporting)

OCI Worker

With original baseimage

Arguments for buildkitd: --oci-worker-snapshotter=overlayfs
Context for buildctl: ./script/buildkit/sample_org

real	0m16.974s
user	0m0.018s
sys	0m0.018s

With stargz baseimage

Arguments for buildkitd: --oci-worker-snapshotter=stargz --oci-worker-proxy-snapshotter-address=/run/containerd-stargz-grpc/containerd-stargz-grpc.sock
Context for buildctl: ./script/buildkit/sample_sgz

containerd Worker

real 0m3.044s
user 0m0.011s
sys 0m0.011s

With original baseimage

Arguments for buildkitd: --oci-worker=false --containerd-worker=true
Context for buildctl: ./script/buildkit/sample_org

real 0m17.914s
user 0m0.029s
sys 0m0.017s

With stargz baseimage

Arguments for buildkitd: --oci-worker=false --containerd-worker=true --containerd-worker-snapshotter=stargz
Context for buildctl: ./script/buildkit/sample_sgz

real 0m2.976s
user 0m0.000s
sys 0m0.018s

Buildkit integration performance(with exporting: --output type=tar)

OCI Worker

With original baseimage

Arguments for buildkitd: --oci-worker-snapshotter=overlayfs
Context for buildctl: ./script/buildkit/sample_org

real	0m24.823s
user	0m4.136s
sys	0m1.756s

With stargz baseimage

Arguments for buildkitd: --oci-worker-snapshotter=stargz --oci-worker-proxy-snapshotter-address=/run/containerd-stargz-grpc/containerd-stargz-grpc.sock
Context for buildctl: ./script/buildkit/sample_sgz

real	1m35.111s
user	0m11.298s
sys	0m6.532s

containerd Worker

With original baseimage

Arguments for buildkitd: --oci-worker=false --containerd-worker=true
Context for buildctl: ./script/buildkit/sample_org

real	0m34.437s
user	0m4.173s
sys	0m1.891s

With stargz baseimage

Arguments for buildkitd: --oci-worker=false --containerd-worker=true --containerd-worker-snapshotter=stargz
Context for buildctl: ./script/buildkit/sample_sgz

real	1m26.735s
user	0m10.159s
sys	0m6.171s

[github-actions]

Benchmarking Result

Runs on the ubuntu-18.04 runner on Github Actions.

python:3.7

mode	pull(sec)	create(sec)	run(sec)
legacy	31.67469096183777	0.0321660041809082	0.15041399002075195
stargz	2.2904369831085205	0.06085491180419922	1.0506479740142822
estargz	2.5134129524230957	0.1449909210205078	0.17807698249816895

gcc:9.2.0

mode	pull(sec)	create(sec)	run(sec)
legacy	32.86303901672363	0.0327608585357666	0.13144803047180176
stargz	2.5307531356811523	0.04351210594177246	2.359808921813965
estargz	2.3148438930511475	0.04931497573852539	2.002725124359131

rethinkdb:2.3.6

mode	pull(sec)	create(sec)	run(sec)
legacy	8.167922973632812	0.040946006774902344	0.14784908294677734
stargz	1.871750831604004	0.05150914192199707	1.833923101425171
estargz	1.6254642009735107	1.3515660762786865	0.19742989540100098

glassfish:4.1-jdk8

mode	pull(sec)	create(sec)	run(sec)
legacy	27.817287921905518	0.026812076568603516	2.305588960647583
stargz	2.4905929565429688	0.04720902442932129	14.54265809059143
estargz	2.451666831970215	7.0523388385772705	2.938326120376587

[github-actions]

Buildkit integration performance(without exporting)

OCI Worker

With original baseimage

Arguments for buildkitd: --oci-worker-snapshotter=overlayfs
Context for buildctl: ./script/buildkit/sample_org

real	0m16.473s
user	0m0.024s
sys	0m0.012s

With stargz baseimage

Arguments for buildkitd: --oci-worker-snapshotter=stargz --oci-worker-proxy-snapshotter-address=/run/containerd-stargz-grpc/containerd-stargz-grpc.sock
Context for buildctl: ./script/buildkit/sample_sgz

real	0m3.054s
user	0m0.015s
sys	0m0.007s

containerd Worker

With original baseimage

Arguments for buildkitd: --oci-worker=false --containerd-worker=true
Context for buildctl: ./script/buildkit/sample_org

real	0m17.628s
user	0m0.016s
sys	0m0.032s

With stargz baseimage

Arguments for buildkitd: --oci-worker=false --containerd-worker=true --containerd-worker-snapshotter=stargz
Context for buildctl: ./script/buildkit/sample_sgz

real	0m2.944s
user	0m0.006s
sys	0m0.012s

Buildkit integration performance(with exporting: --output type=tar)

OCI Worker

With original baseimage

Arguments for buildkitd: --oci-worker-snapshotter=overlayfs
Context for buildctl: ./script/buildkit/sample_org

real	0m25.007s
user	0m4.507s
sys	0m1.571s

With stargz baseimage

Arguments for buildkitd: --oci-worker-snapshotter=stargz --oci-worker-proxy-snapshotter-address=/run/containerd-stargz-grpc/containerd-stargz-grpc.sock
Context for buildctl: ./script/buildkit/sample_sgz

real	1m31.731s
user	0m9.909s
sys	0m6.174s

containerd Worker

With original baseimage

Arguments for buildkitd: --oci-worker=false --containerd-worker=true
Context for buildctl: ./script/buildkit/sample_org

real	0m31.812s
user	0m4.004s
sys	0m1.555s

With stargz baseimage

Arguments for buildkitd: --oci-worker=false --containerd-worker=true --containerd-worker-snapshotter=stargz
Context for buildctl: ./script/buildkit/sample_sgz

real	1m23.942s
user	0m9.094s
sys	0m5.968s

[AkihiroSuda]

Can you open a proposal for BuildKit changes in https://github.com/moby/buildkit/issues

cc @tonistiigi

[ktock]

Yes. Before that, please let me re-open this PR from my forked repo.

[ktock]

Reopened on #53.