Running ./hack/update-openapi-spec.sh leaves api/openapi-spec/swagger.json file empty

[xing-yang]

What happened:
After running ./hack/update-openapi-spec.sh with the latest code, api/openapi-spec/swagger.json file became empty even when no changes were made in the code.

What you expected to happen:
After running ./hack/update-openapi-spec.sh with the latest code, api/openapi-spec/swagger.json file should not be changed when no changes were made in the code.

When I tried the same thing a few week back, I didn't encounter this problem.

How to reproduce it (as minimally and precisely as possible):
Download the latest kubernetes/kubernetes code and run the following script to update openapi spec.

# ./hack/update-openapi-spec.sh

make: Entering directory '/root/Github/kubernetes'
make[1]: Entering directory '/root/Github/kubernetes'
stat /root/Github/kubernetes/_output/local/Github/kubernetes/third_party/etcd-v3.3.15-linux-amd64/Documentation/README.md: no such file or directory
make[1]: Leaving directory '/root/Github/kubernetes'
+++ [1103 02:32:07] Building go targets for linux/amd64:
    cmd/kube-apiserver
make: Leaving directory '/root/Github/kubernetes'
etcd --advertise-client-urls http://127.0.0.1:2379 --data-dir /tmp/tmp.EcaRTaU7fn --listen-client-urls http://127.0.0.1:2379 --log-level=debug > "/dev/null" 2>/dev/null
Waiting for etcd to come up.
+++ [1103 02:34:36] On try 1, etcd: : {"health":"true"}
{"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"2","raft_term":"2"}}+++ [1103 02:34:36] Starting kube-apiserver
+++ [1103 02:34:47] On try 7, apiserver: : ok
+++ [1103 02:34:47] Updating
    ./hack/../api/openapi-spec
+++ [1103 02:34:48] Clean up complete

api/openapi-spec/swagger.json became empty after this.

Anything else we need to know?:
There's a line in update-openapi-spec.sh that runs a curl command curl -w "\n" -kfs --oauth2-bearer dummy_token https://127.0.0.1:8050/openapi/v2.

This line was changed a few days ago by this commit: cfc2b33
PR: #95856

I checked out the commit before cfc2b33 and ran /update-openapi-spec.sh. It works. swagger.json is not empty any more.

# git checkout eb63da77ea59af34033feb39bc8ccb5ce57c14a0
Note: switching to 'eb63da77ea59af34033feb39bc8ccb5ce57c14a0'.

# ./hack/update-openapi-spec.sh
make: Entering directory '/root/Github/kubernetes'
make[1]: Entering directory '/root/Github/kubernetes'
+++ [1103 02:49:46] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/conversion-gen
make[1]: Leaving directory '/root/Github/kubernetes'
+++ [1103 02:50:25] Building go targets for linux/amd64:
    cmd/kube-apiserver
make: Leaving directory '/root/Github/kubernetes'
etcd --advertise-client-urls http://127.0.0.1:2379 --data-dir /tmp/tmp.fyS4an1E6o --listen-client-urls http://127.0.0.1:2379 --log-level=debug > "/dev/null" 2>/dev/null
Waiting for etcd to come up.
+++ [1103 02:51:47] On try 1, etcd: : {"health":"true"}
{"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"2","raft_term":"2"}}+++ [1103 02:51:47] Starting kube-apiserver
+++ [1103 02:51:51] On try 5, apiserver: : ok
+++ [1103 02:51:51] Updating
    ./hack/../api/openapi-spec
+++ [1103 02:51:51] SUCCESS
+++ [1103 02:51:53] Clean up complete

Is there any new setting required before running ./hack/update-openapi-spec.sh now?

Environment:

• Kubernetes version (use kubectl version):

# kubectl version
Client Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.0-beta.0.127+dd56138e55d11c-dirty", GitCommit:"dd56138e55d11c557bd5aebe5fcd4b68eb7c4191", GitTreeState:"dirty", BuildDate:"2020-11-01T02:55:09Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

[xing-yang]

/sig api-machinery

[liggitt]

cc @knight42

[knight42]

@xing-yang Hi, thanks for the report, but it is a bit weird that I could not reproduce locally:

$ git log -1
commit f7cfe3ba0f0434f82208cc940c0770984e5758f5 (HEAD -> master, upstream/master)
Merge: 5beb14f0453 625e47aaa27
Author: Kubernetes Prow Robot <k8s-ci-robot@users.noreply.github.com>
Date:   Tue Nov 3 05:50:15 2020 -0800

    Merge pull request #95076 from dougsland/deprecate-delete-local-data

    kubectl: deprecate --delete-local-data

$ ./hack/update-openapi-spec.sh
+++ [1103 23:29:08] Building go targets for darwin/amd64:
    cmd/kube-apiserver
etcd --advertise-client-urls http://127.0.0.1:2379 --data-dir /var/folders/bp/20yxytt90055krc_0hct5qrm0000gn/T/tmp.SYdoXE3NRq --listen-client-urls http://127.0.0.1:2379 --log-level=debug > "/dev/null" 2>/dev/null
Waiting for etcd to come up.
+++ [1103 23:29:29] On try 2, etcd: : {"health":"true"}
{"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"2","raft_term":"2"}}+++ [1103 23:29:29] Starting kube-apiserver
+++ [1103 23:29:36] On try 7, apiserver: : ok
+++ [1103 23:29:36] Updating
    ./hack/../api/openapi-spec
+++ [1103 23:29:37] SUCCESS
+++ [1103 23:29:38] Clean up complete

$ wc -c api/openapi-spec/swagger.json
5286263 api/openapi-spec/swagger.json

and besides, ./hack/verify-openapi-spec.sh completes successfully on my machine.

@liggitt Could you reproduce?

[xing-yang]

@knight42 Someone else also ran into this same problem and asked a question on sig-api-machinery slack yesterday. So I'm not the only one who ran into this.
https://kubernetes.slack.com/archives/C0EG7JC6T/p1604344853456100

[knight42]

@xing-yang could you change the args passed to curl in ./hack/update-openapi-spec.sh to curl -w "\n" -kfsS ... and rerun the script to see if there is any error?

[knight42]

since it is already late here(UTC+8), I would check this issue again tomorrow 😴

[janosi]

@knight42 I was the "someone else" who ran into this problem :) I changed the script to use curl -w "\n" -kfsS... The result:

hack/update-openapi-spec.sh 
make: Entering directory '/home/ubuntu/go/src/k8s.io/kubernetes'
make[1]: Entering directory '/home/ubuntu/go/src/k8s.io/kubernetes'
make[1]: Leaving directory '/home/ubuntu/go/src/k8s.io/kubernetes'
+++ [1103 18:53:58] Building go targets for linux/amd64:
    cmd/kube-apiserver
make: Leaving directory '/home/ubuntu/go/src/k8s.io/kubernetes'
etcd --advertise-client-urls http://127.0.0.1:2379 --data-dir /tmp/tmp.77sutYIQw8 --listen-client-urls http://127.0.0.1:2379 --log-level=debug > "/dev/null" 2>/dev/null
Waiting for etcd to come up.
+++ [1103 18:54:15] On try 2, etcd: : {"health":"true"}
{"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"2","raft_term":"2"}}+++ [1103 18:54:15] Starting kube-apiserver
+++ [1103 18:54:26] On try 8, apiserver: : ok
+++ [1103 18:54:26] Updating 
    hack/../api/openapi-spec
curl: (22) The requested URL returned error: 403 
+++ [1103 18:54:27] Clean up complete

Actually, it was 403 without the "S", too, but that is visible only if curl is executed in verbose mode.

[knight42]

@janosi Hi, thanks for your information. Given that the apiserver returned 403, I suspect the flag --oauth2-bearer passed to curl has no effect in your environment, would you mind posting the output of curl --version? I am using curl 7.64.1:

$ curl --version
curl 7.64.1 (x86_64-apple-darwin19.0) libcurl/7.64.1 (SecureTransport) LibreSSL/2.8.3 zlib/1.2.11 nghttp2/1.39.2
Release-Date: 2019-03-27
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL UnixSockets

In addition, I think you could try changing the curl command in hack/update-openapi-sepc.sh to the following:

curl -w "\n" -kfsS -H 'Authorization: Bearer dummy_token' "https://${API_HOST}:${API_PORT}/openapi/v2" | jq -S '.info.version="unversioned"' > "${OPENAPI_ROOT_DIR}/swagger.json"

and see if the problem is solved.

[xing-yang]

@knight42 I tried the following and it worked for me. swagger.json is no longer empty.

curl -w "\n" -kfsS -H 'Authorization: Bearer dummy_token' "https://${API_HOST}:${API_PORT}/openapi/v2" | jq -S '.info.version="unversioned"' > "${OPENAPI_ROOT_DIR}/swagger.json"

curl version:

# curl --version
curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.1 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
Release-Date: 2018-01-24
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL

[knight42]

@xing-yang Thanks for your confirmation, and I have filed #96204 to address this issue.

[janosi]

@knight42 I can confirm, it works with the changed curl command.

curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.1 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
Release-Date: 2018-01-24
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL 

[fedebongio]

/triage accepted