github-assets-uploader very often fails

[chrislusf]

Currently the upload very often fails at github-assets-uploader. (Previously it was working fine.)

https://github.com/chrislusf/seaweedfs/runs/4107363083?check_suite_focus=true

Sometimes it can run partially after serveral retries: https://github.com/chrislusf/seaweedfs/actions/runs/1422094429

+ tar cvfz weed-large-disk-20211104-1601-linux-amd64.tar.gz weed-large-disk
weed-large-disk
++ md5sum weed-large-disk-20211104-1601-linux-amd64.tar.gz
++ cut -d ' ' -f 1
+ MD5_SUM=fd5586ddecc22fd85307d9081bec4a12
++ sha256sum weed-large-disk-20211104-1601-linux-amd64.tar.gz
++ cut -d ' ' -f 1
+ SHA256_SUM=71e73515078143412eb9db170d6a28d48f3a82cbe854d6e7fcfa9971b173da94
+ GITHUB_ASSETS_UPLOADR_EXTRA_OPTIONS=
+ '[' TRUE == TRUE ']'
+ GITHUB_ASSETS_UPLOADR_EXTRA_OPTIONS=-overwrite
+ github-assets-uploader -f weed-large-disk-20211104-1601-linux-amd64.tar.gz -mediatype application/gzip -overwrite -repo chrislusf/seaweedfs -token *** -tag dev
Post "https://uploads.github.com/repos/chrislusf/seaweedfs/releases/52499161/assets?name=weed-large-disk-20211104-1601-linux-amd64.tar.gz": unexpected EOF

[wangyoucao577]

It seems similar with #62.
How big does your asset? Is it occur often on big assets?

[wangyoucao577]

I made a branch that upgrades underlying github package. If it caused by Github's issue, it maybe helpful. Could you please help to have a try? wangyoucao577/go-release-action@bugfix/upload-fail. Thanks!

[chrislusf]

tried just now with the same result. https://github.com/chrislusf/seaweedfs/runs/4125792435?check_suite_focus=true

The binary size is about 33MB.

[chrislusf]

Maybe add some delays between each option, so the binary can be read successfully in order to upload.

This is rather serious since this action can only be successful if all the uploads succeed. For a multi-OS multi-CPU build with many variations, this would almost never succeed. I have changed the release process to have action file for one OS, but still need to try a lot of retries.

[chrislusf]

How about trying other upload scripts besides "github-assets-uploader"?

[wangyoucao577]

I have added retrying max to 3 times if upload fail on the branch, please have a look whether it has improvment.

[chrislusf]

Here is the retry output from this run: https://github.com/chrislusf/seaweedfs/runs/4155150295?check_suite_focus=true

+ tar cvfz weed-large-disk-20211109-1651-linux-amd64.tar.gz weed-large-disk
weed-large-disk
++ md5sum weed-large-disk-20211109-1651-linux-amd64.tar.gz
++ cut -d ' ' -f 1
+ MD5_SUM=3ae054d90cfd65c1a4856471ca24d113
++ sha256sum weed-large-disk-20211109-1651-linux-amd64.tar.gz
++ cut -d ' ' -f 1
+ SHA256_SUM=d0208ce13fd85ecfde0464383de1f143d342a178463d64bbd94500174f95ecc3
+ GITHUB_ASSETS_UPLOADR_EXTRA_OPTIONS=
+ '[' TRUE == TRUE ']'
+ GITHUB_ASSETS_UPLOADR_EXTRA_OPTIONS=-overwrite
+ github-assets-uploader -f weed-large-disk-20211109-1651-linux-amd64.tar.gz -mediatype application/gzip -overwrite -repo chrislusf/seaweedfs -token *** -tag dev -retry 3
Upload asset error: Post "https://uploads.github.com/repos/chrislusf/seaweedfs/releases/52499161/assets?name=weed-large-disk-20211109-1651-linux-amd64.tar.gz": unexpected EOF, will retry later
Upload asset error: Post "https://uploads.github.com/repos/chrislusf/seaweedfs/releases/52499161/assets?name=weed-large-disk-20211109-1651-linux-amd64.tar.gz": http2: Transport: cannot retry err [stream error: stream ID 1; REFUSED_STREAM] after Request.Body was written; define Request.GetBody to avoid this error, will retry later
Post "https://uploads.github.com/repos/chrislusf/seaweedfs/releases/52499161/assets?name=weed-large-disk-20211109-1651-linux-amd64.tar.gz": http2: Transport: cannot retry err [stream error: stream ID 3; REFUSED_STREAM] after Request.Body was written; define Request.GetBody to avoid this error

[wangyoucao577]

@chrislusf Retrying doesn't work, there must be something wrong in the underlying. After some investigation, I think it's a bug of net/http which seems have been resolved several years ago, not sure why happen again. See golang/go#20985 and https://stackoverflow.com/questions/44089111/random-refused-stream-making-a-post-multipart-form-request.

I have updated the branch with upgrading the oauth2 package and use latest go to build the binary. If it still happens, please follow https://pkg.go.dev/net/http#Client to try to disable the http2 by export the env GODEBUG=http2client=0.

[chrislusf]

Added https://github.com/chrislusf/seaweedfs/blob/master/.github/workflows/binaries_dev.yml#L40
This is the output:
https://github.com/chrislusf/seaweedfs/runs/4168981192?check_suite_focus=true

[chrislusf]

btw: You can create a PR and make changes to this binaries_dev.yml file. Each change should trigger the github action to run. This should help to debug and iterate faster.

[wangyoucao577]

@chrislusf
seaweedfs/seaweedfs#2440

2 workflows awaiting approval First-time contributors need a maintainer to approve running workflows. Learn more.

[chrislusf]

Sorry, I found the PR could not trigger this action since it needs github token to delete resources. I merged your changes. Here is the latest output for https://github.com/chrislusf/seaweedfs/runs/4182578863?check_suite_focus=true

Seems the retry failed because of a partially uploaded file.

+ ls -lha
total 79M
drwxr-xr-x  2 root root 4.0K Nov 11 20:12 .
drwxr-xr-x 29 1001  121 4.0K Nov 11 20:10 ..
-rwxr-xr-x  1 root root  79M Nov 11 20:12 weed-large-disk
+ RELEASE_ASSET_EXT=.tar.gz
+ MEDIA_TYPE=application/gzip
+ '[' darwin == windows ']'
+ shopt -s dotglob
+ tar cvfz weed-large-disk-20211111-2010-darwin-arm64.tar.gz weed-large-disk
weed-large-disk
++ md5sum weed-large-disk-20211111-2010-darwin-arm64.tar.gz
++ cut -d ' ' -f 1
+ MD5_SUM=b2d6f9cdbaef1a128ca8c497c2978c66
++ sha256sum weed-large-disk-20211111-2010-darwin-arm64.tar.gz
++ cut -d ' ' -f 1
+ SHA256_SUM=ee2578024083e902681427816ebaab5adbe1422479db7edcffda21c845f43f07
+ GITHUB_ASSETS_UPLOADR_EXTRA_OPTIONS=
+ '[' TRUE == TRUE ']'
+ GITHUB_ASSETS_UPLOADR_EXTRA_OPTIONS=-overwrite
+ github-assets-uploader -f weed-large-disk-20211111-2010-darwin-arm64.tar.gz -mediatype application/gzip -overwrite -repo chrislusf/seaweedfs -token *** -tag dev -retry 3
Upload asset error: Post "https://uploads.github.com/repos/chrislusf/seaweedfs/releases/52499161/assets?name=weed-large-disk-20211111-2010-darwin-arm64.tar.gz": write tcp 172.17.0.2:35860->140.82.113.13:443: write: connection reset by peer, will retry later
Upload asset error: Post "https://uploads.github.com/repos/chrislusf/seaweedfs/releases/52499161/assets?name=weed-large-disk-20211111-2010-darwin-arm64.tar.gz": http2: Transport: cannot retry err [stream error: stream ID 1; REFUSED_STREAM; received from peer] after Request.Body was written; define Request.GetBody to avoid this error, will retry later
POST https://uploads.github.com/repos/chrislusf/seaweedfs/releases/52499161/assets?name=weed-large-disk-20211111-2010-darwin-arm64.tar.gz: 422 Validation Failed [{Resource:ReleaseAsset Field:name Code:already_exists Message:}]

[wangyoucao577]

@chrislusf I saw there still some use of closed network connection happens. I have updated the branch again to enhance it. Let's take a look whether still error occurs...

[chrislusf]

thanks for monitoring this! It seems much stable now.

[chrislusf]

thanks! I consider this is fixed. I tried with multiple cases. The one using the branch always works now. Thanks!

[wangyoucao577]

@chrislusf Glad to hear that! The branch will be removed soon, please switch to the new release wangyoucao577/go-release-action@v1.21!

[jritsema]

Hello. I'm also experiencing issues. Example here. Anyone have a good solution?

http2: Transport: cannot retry err [stream error: stream ID 1; REFUSED_STREAM; received from peer] after Request.Body was written; define Request.GetBody to avoid this error