Skip to content
This repository has been archived by the owner on Feb 7, 2024. It is now read-only.

Git checkout takes too long in VM #55

Closed
gowthamchinna opened this issue Aug 18, 2020 · 23 comments
Closed

Git checkout takes too long in VM #55

gowthamchinna opened this issue Aug 18, 2020 · 23 comments
Assignees
@mukul-seagate11
Labels
bug Bug in script or CI workflow build failure Build failure due to CI scripts or infrastructure issue.

Comments

@gowthamchinna
Copy link
Contributor

gowthamchinna commented Aug 18, 2020 •
edited
Loading

Git checkout takes too long ( ~15 min ) in cloudform VM.

Priority : High

Observation

  • This issue can be reproducible in the time-frame of 6 AM - 10 AM IST (Not sure when it exactly starts).
  • This issue only occurs in the Cloudform VM
Checkout Log in Citrix VDI

  • Repo : cortx-csm

  • Timetaken : 0.23 m

Network Speed

curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -
Retrieving speedtest.net configuration...
Testing from Seagate Technology LLC (192.55.14.36)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Singtel (Singapore) [6.13 km]: 5.158 ms
Testing download speed................................................................................
Download: 642.32 Mbit/s
Testing upload speed......................................................................................................
Upload: 767.29 Mbit/s

Git Checkout Log

$ time GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone https://<PAT>@github.com/Seagate/cortx-cs
m.git
07:55:44.667551 exec-cmd.c:238          trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
07:55:44.670553 git.c:439               trace: built-in: git clone https://<PAT>@github.com/Seagate/cortx-csm.git
Cloning into 'cortx-csm'...
07:55:45.094549 run-command.c:663       trace: run_command: git remote-https origin https://<PAT>@github.com/Seagate/cortx-csm.git
07:55:45.134548 exec-cmd.c:238          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
07:55:45.141547 git.c:704               trace: exec: git-remote-https origin https://<PAT>@github.com/Seagate/cortx-csm.git
07:55:45.141547 run-command.c:663       trace: run_command: git-remote-https origin https://<PAT>@github.com/Seagate/cortx-csm.git
07:55:45.204547 exec-cmd.c:238          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
* Couldn't find host github.com in the _netrc file; using defaults
*   Trying 52.74.223.119:443...
* TCP_NODELAY set
* Connected to github.com (52.74.223.119) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=github.com
*  start date: May  5 00:00:00 2020 GMT
*  expire date: May 10 12:00:00 2022 GMT
*  subjectAltName: host "github.com" matched cert's "github.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
*  SSL certificate verify ok.
> GET /Seagate/cortx-csm.git/info/refs?service=git-upload-pack HTTP/1.1
Host: github.com
User-Agent: git/2.22.0.windows.1
Accept: */*
Accept-Encoding: deflate, gzip
Pragma: no-cache

* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Authorization Required
< Server: GitHub Babel 2.0
< Content-Type: text/plain
< Content-Length: 21
< WWW-Authenticate: Basic realm="GitHub"
< X-Frame-Options: DENY
< X-GitHub-Request-Id: 5824:63DC:2DDF87A:3EB5973:5F39EAA9
<
* Ignoring the response-body
* Connection #0 to host github.com left intact
* Issue another request to this URL: 'https://<PAT>@github.com/Seagate/cortx-csm.git/info/refs?service=git-upload-pack'
* Couldn't find host github.com in the _netrc file; using defaults
* Found bundle for host github.com: 0x2dee370 [serially]
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (52.74.223.119) port 443 (#0)
* Server auth using Basic with user '<PAT>'
> GET /Seagate/cortx-csm.git/info/refs?service=git-upload-pack HTTP/1.1
Host: github.com
Authorization: Basic ZTM5ZDI0MGIyYmI1NWE3MzkxOTMxOTg1ZDAyMDFmNzQ5ODk2NzdmNTo=
User-Agent: git/2.22.0.windows.1
Accept: */*
Accept-Encoding: deflate, gzip
Pragma: no-cache

* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-advertisement
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-Frame-Options: DENY
< X-GitHub-Request-Id: 5824:63DC:2DDF8E0:3EB59EF:5F39EAAA
<
* Connection #0 to host github.com left intact
07:55:47.600703 run-command.c:663       trace: run_command: git fetch-pack --stateless-rpc --stdin --lock-pack --thin --check-self-contained-and-connected --cloning https://<PAT>@github.com/Seagate/cortx-csm.git/
07:55:47.659702 exec-cmd.c:238          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
07:55:47.668701 git.c:439               trace: built-in: git fetch-pack --stateless-rpc --stdin --lock-pack --thin --check-self-contained-and-connected --cloning https://<PAT>@github.com/Seagate/cortx-csm.git/
* Couldn't find host github.com in the _netrc file; using defaults
* Found bundle for host github.com: 0x2dee370 [serially]
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (52.74.223.119) port 443 (#0)
* Server auth using Basic with user '<PAT>'
> POST /Seagate/cortx-csm.git/git-upload-pack HTTP/1.1
Host: github.com
Authorization: Basic ZTM5ZDI0MGIyYmI1NWE3MzkxOTMxOTg1ZDAyMDFmNzQ5ODk2NzdmNTo=
User-Agent: git/2.22.0.windows.1
Accept-Encoding: deflate, gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Encoding: gzip
Content-Length: 6224

* upload completely sent off: 6224 out of 6224 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-result
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-Frame-Options: DENY
< X-GitHub-Request-Id: 5824:63DC:2DDF9FA:3EB5AF9:5F39EAAB
<
remote: Enumerating objects: 291, done.
remote: Counting objects: 100% (291/291), done.
remote: Compressing objects: 100% (213/213), done.
07:55:49.360686 run-command.c:663       trace: run_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 22760 on sgspictvw2228' --check-self-contained-and-connected --pack_header=2,56013
07:55:49.435685 exec-cmd.c:238          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
07:55:49.472199 git.c:439               trace: built-in: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 22760 on sgspictvw2228' --check-self-contained-and-connected --pack_header=2,56013
* Connection #0 to host github.com left intactiB | 8.96 MiB/s
remote: Total 56013 (delta 132), reused 143 (delta 72), pack-reused 55722 eceiving objects: 100% (56013/56013), 40.56 MiB | 8Receiving objects: 100% (56013/56013), 41.69 MiB | 8.09 MiB/s, done.

Resolving deltas: 100% (25949/25949), done.
07:55:56.126192 run-command.c:663       trace: run_command: git rev-list --objects --stdin --not --all --quiet '--progress=Checking connectivity'
07:55:56.173192 exec-cmd.c:238          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
07:55:56.183192 git.c:439               trace: built-in: git rev-list --objects --stdin --not --all --quiet '--progress=Checking connectivity'
Checking out files: 100% (686/686), done.

real    0m23.450s
user    0m0.000s
sys     0m0.031s

Checkout Log in Cloudform VM

  • Repo : cortx-csm

  • Timetaken : 14.17 m

Network Speed

[root@ssc-vm-0034 736373]# curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -
Retrieving speedtest.net configuration...
Testing from Seagate Technology LLC (134.204.222.36)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Nitel (Denver, CO) [48.60 km]: 35.627 ms
Testing download speed................................................................................
Download: 198.21 Mbit/s
Testing upload speed................................................................................................
Upload: 310.32 Mbit/s

Git Checkout Log

[root@4be3b7e549a9 /]# time GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone https://<PAT>@github.com/Seagate/cortx-csm.git
02:32:13.129992 git.c:371               trace: built-in: git 'clone' 'https://<PAT>@github.com/Seagate/cortx-csm.git'
Cloning into 'cortx-csm'...
02:32:13.133696 run-command.c:369       trace: run_command: 'git-remote-https' 'origin' 'https://<PAT>@github.com/Seagate/cortx-csm.git'
* Couldn't find host github.com in the .netrc file; using defaults
* About to connect() to github.com port 443 (#0)
*   Trying 140.82.114.4...
* Connected to github.com (140.82.114.4) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=github.com,O="GitHub, Inc.",L=San Francisco,ST=California,C=US
*       start date: May 05 00:00:00 2020 GMT
*       expire date: May 10 12:00:00 2022 GMT
*       common name: github.com
*       issuer: CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US
> GET /Seagate/cortx-csm.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/2.12.2
Host: github.com
Accept: */*
Accept-Encoding: gzip
Accept-Language: en-US, *;q=0.9
Pragma: no-cache

< HTTP/1.1 401 Authorization Required
< Server: GitHub Babel 2.0
< Content-Type: text/plain
< Content-Length: 21
< WWW-Authenticate: Basic realm="GitHub"
< X-GitHub-Request-Id: 2A98:2BC3:3987949:54AED2F:5F39EC2D
< X-Frame-Options: DENY
<
* Ignoring the response-body
* Connection #0 to host github.com left intact
* Issue another request to this URL: 'https://<PAT>@github.com/Seagate/cortx-csm.git/info/refs?service=git-upload-pack'
* Couldn't find host github.com in the .netrc file; using defaults
* Found bundle for host github.com: 0xad4a60
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (140.82.114.4) port 443 (#0)
* Server auth using Basic with user '<PAT>'
> GET /Seagate/cortx-csm.git/info/refs?service=git-upload-pack HTTP/1.1
Authorization: Basic ZTM5ZDI0MGIyYmI1NWE3MzkxOTMxOTg1ZDAyMDFmNzQ5ODk2NzdmNTo=
User-Agent: git/2.12.2
Host: github.com
Accept: */*
Accept-Encoding: gzip
Accept-Language: en-US, *;q=0.9
Pragma: no-cache

< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-advertisement
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: 2A98:2BC3:39879CA:54AED5C:5F39EC2D
< X-Frame-Options: DENY
<
* Connection #0 to host github.com left intact
02:32:15.583453 run-command.c:369       trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://<PAT>@github.com/Seagate/cortx-csm.git/'
02:32:15.584471 exec_cmd.c:118          trace: exec: 'git' 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://<PAT>@github.com/Seagate/cortx-csm.git/'
02:32:15.586117 git.c:371               trace: built-in: git 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://<PAT>@github.com/Seagate/cortx-csm.git/'
* Couldn't find host github.com in the .netrc file; using defaults
* Found bundle for host github.com: 0xad4a60
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (140.82.114.4) port 443 (#0)
* Server auth using Basic with user '<PAT>'
> POST /Seagate/cortx-csm.git/git-upload-pack HTTP/1.1
Authorization: Basic ZTM5ZDI0MGIyYmI1NWE3MzkxOTMxOTg1ZDAyMDFmNzQ5ODk2NzdmNTo=
User-Agent: git/2.12.2
Host: github.com
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Encoding: gzip
Content-Length: 6214

* upload completely sent off: 6214 out of 6214 bytes
< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-result
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: 2A98:2BC3:3987A2E:54AEE1E:5F39EC2E
< X-Frame-Options: DENY
<
remote: Enumerating objects: 291, done.
remote: Counting objects: 100% (291/291), done.
remote: Compressing objects: 100% (213/213), done.
02:32:16.435083 run-command.c:369       trace: run_command: 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 53 on 4be3b7e549a9' '--check-self-contained-and-connected' '--pack_header=2,56013'
02:32:16.436121 exec_cmd.c:118          trace: exec: 'git' 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 53 on 4be3b7e549a9' '--check-self-contained-and-connected' '--pack_header=2,56013'
02:32:16.437700 git.c:371               trace: built-in: git 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 53 on 4be3b7e549a9' '--check-self-contained-and-connected' '--pack_header=2,56013'
* Connection #0 to host github.com left intactiB | 74.00 KiB/s
remote: Total 56013 (delta 132), reused 143 (delta 72), pack-reused 55722
Receiving objects: 100% (56013/56013), 41.69 MiB | 68.00 KiB/s, done.
Resolving deltas: 100% (25949/25949), done.
02:46:30.154135 run-command.c:369       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'
02:46:30.154769 exec_cmd.c:118          trace: exec: 'git' 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'
02:46:30.156481 git.c:371               trace: built-in: git 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'

real    14m17.110s
user    0m3.559s
sys     0m1.760s

@gowthamchinna gowthamchinna added bug Bug in script or CI workflow build failure Build failure due to CI scripts or infrastructure issue. labels Aug 18, 2020
@shailesh-vaidya
Copy link
Contributor

@mukul-seagate11 We face this issue almost daily in Jenkins build process. Can we raise ticket with GitHub for this.

@mukul-seagate11
Copy link
Contributor

@mukul-seagate11 We face this issue almost daily in Jenkins build process. Can we raise ticket with GitHub for this.

Sure, we can take this forward with GitHub call

@stale
Copy link

stale bot commented Aug 25, 2020

This issue/pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the auto-stale label Aug 25, 2020
@shailesh-vaidya
Copy link
Contributor

@mukul-seagate11 As updated by Dhruv in today's call, Can you please raise support ticket for this with GitHub team.

@stale stale bot removed the auto-stale label Aug 25, 2020
@mukul-seagate11
Copy link
Contributor

@mukul-seagate11 As updated by Dhruv in today's call, Can you please raise support ticket for this with GitHub team.

OK sure

@mukul-seagate11
Copy link
Contributor

image

@gowthamchinna
Copy link
Contributor Author

Absorbing random failures in pipeline

http://eos-jenkins.colo.seagate.com/job/Cortx-Dev/job/RHEL-7.7/job/NFS/53/

http://eos-jenkins.colo.seagate.com/job/Cortx-Dev/job/RHEL-7.7/job/S3server/155/console

@gowthamchinna
Copy link
Contributor Author

+1

http://eos-jenkins.colo.seagate.com/job/Cortx-Dev/job/RHEL-7.7/job/Hare/160/console

@gowthamchinna
Copy link
Contributor Author

gowthamchinna commented Aug 26, 2020 •
edited
Loading

+1
http://eos-jenkins.colo.seagate.com/job/Cortx-Dev/job/RHEL-7.7/job/Motr/115//console

image

This issue occurred between 25 th Aug 11 PM IST to 26 th Aug 2.20 AM IST

@gowthamchinna
Copy link
Contributor Author

Update : The checkout is working on the above mentioned time frame, But still getting random failure on checkout

@mukul-seagate11
Copy link
Contributor

Casey Mokricky (GitHub Support)
Aug 25, 2020, 4:16 PM UTC

Hi Mukul,

Thank you for contacting GitHub Enterprise Support.

Just to make sure I'm understanding correctly, it sounds like your git client is having performance issues on a CloudForm VM instance. Is that correct?

If so, please prefix your git command with the GIT_TRACE_PERFORMANCE environment variable and respond with any output. For example:

GIT_TRACE_PERFORMANCE=1 git clone https://github.com/my/repo.git
Feel free to include any additional information as well.

Thank you,
Casey

@shailesh-vaidya
Copy link
Contributor

@mukul-seagate11 - We are observing this issue quite frequently now. @gowthamchinna has collected debug logs as below. Git checkout is taking around 14 mins which is causing build failures.

[root@4be3b7e549a9 /]# time GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone https://<PAT>@github.com/Seagate/cortx-csm.git
02:32:13.129992 git.c:371               trace: built-in: git 'clone' 'https://<PAT>@github.com/Seagate/cortx-csm.git'
Cloning into 'cortx-csm'...
02:32:13.133696 run-command.c:369       trace: run_command: 'git-remote-https' 'origin' 'https://<PAT>@github.com/Seagate/cortx-csm.git'
* Couldn't find host github.com in the .netrc file; using defaults
* About to connect() to github.com port 443 (#0)
*   Trying 140.82.114.4...
* Connected to github.com (140.82.114.4) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=github.com,O="GitHub, Inc.",L=San Francisco,ST=California,C=US
*       start date: May 05 00:00:00 2020 GMT
*       expire date: May 10 12:00:00 2022 GMT
*       common name: github.com
*       issuer: CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US
> GET /Seagate/cortx-csm.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/2.12.2
Host: github.com
Accept: */*
Accept-Encoding: gzip
Accept-Language: en-US, *;q=0.9
Pragma: no-cache

< HTTP/1.1 401 Authorization Required
< Server: GitHub Babel 2.0
< Content-Type: text/plain
< Content-Length: 21
< WWW-Authenticate: Basic realm="GitHub"
< X-GitHub-Request-Id: 2A98:2BC3:3987949:54AED2F:5F39EC2D
< X-Frame-Options: DENY
<
* Ignoring the response-body
* Connection #0 to host github.com left intact
* Issue another request to this URL: 'https://<PAT>@github.com/Seagate/cortx-csm.git/info/refs?service=git-upload-pack'
* Couldn't find host github.com in the .netrc file; using defaults
* Found bundle for host github.com: 0xad4a60
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (140.82.114.4) port 443 (#0)
* Server auth using Basic with user '<PAT>'
> GET /Seagate/cortx-csm.git/info/refs?service=git-upload-pack HTTP/1.1
Authorization: Basic ZTM5ZDI0MGIyYmI1NWE3MzkxOTMxOTg1ZDAyMDFmNzQ5ODk2NzdmNTo=
User-Agent: git/2.12.2
Host: github.com
Accept: */*
Accept-Encoding: gzip
Accept-Language: en-US, *;q=0.9
Pragma: no-cache

< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-advertisement
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: 2A98:2BC3:39879CA:54AED5C:5F39EC2D
< X-Frame-Options: DENY
<
* Connection #0 to host github.com left intact
02:32:15.583453 run-command.c:369       trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://<PAT>@github.com/Seagate/cortx-csm.git/'
02:32:15.584471 exec_cmd.c:118          trace: exec: 'git' 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://<PAT>@github.com/Seagate/cortx-csm.git/'
02:32:15.586117 git.c:371               trace: built-in: git 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://<PAT>@github.com/Seagate/cortx-csm.git/'
* Couldn't find host github.com in the .netrc file; using defaults
* Found bundle for host github.com: 0xad4a60
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (140.82.114.4) port 443 (#0)
* Server auth using Basic with user '<PAT>'
> POST /Seagate/cortx-csm.git/git-upload-pack HTTP/1.1
Authorization: Basic ZTM5ZDI0MGIyYmI1NWE3MzkxOTMxOTg1ZDAyMDFmNzQ5ODk2NzdmNTo=
User-Agent: git/2.12.2
Host: github.com
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Encoding: gzip
Content-Length: 6214

* upload completely sent off: 6214 out of 6214 bytes
< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-result
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: 2A98:2BC3:3987A2E:54AEE1E:5F39EC2E
< X-Frame-Options: DENY
<
remote: Enumerating objects: 291, done.
remote: Counting objects: 100% (291/291), done.
remote: Compressing objects: 100% (213/213), done.
02:32:16.435083 run-command.c:369       trace: run_command: 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 53 on 4be3b7e549a9' '--check-self-contained-and-connected' '--pack_header=2,56013'
02:32:16.436121 exec_cmd.c:118          trace: exec: 'git' 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 53 on 4be3b7e549a9' '--check-self-contained-and-connected' '--pack_header=2,56013'
02:32:16.437700 git.c:371               trace: built-in: git 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 53 on 4be3b7e549a9' '--check-self-contained-and-connected' '--pack_header=2,56013'
* Connection #0 to host github.com left intactiB | 74.00 KiB/s
remote: Total 56013 (delta 132), reused 143 (delta 72), pack-reused 55722
Receiving objects: 100% (56013/56013), 41.69 MiB | 68.00 KiB/s, done.
Resolving deltas: 100% (25949/25949), done.
02:46:30.154135 run-command.c:369       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'
02:46:30.154769 exec_cmd.c:118          trace: exec: 'git' 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'
02:46:30.156481 git.c:371               trace: built-in: git 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'

real    14m17.110s
user    0m3.559s
sys     0m1.760s

@shailesh-vaidya
Copy link
Contributor

@mukul-seagate11 @gowthamchinna - Currently all Jenkins jobs are failing at git checkout stage. I suspect this SSC VM specific issue. I have raised https://jts.seagate.com/browse/LRL-391 for the same.
CC: @amolkongre @priyankdal

@kpricepc
Copy link

This appears to be an issue site wide at LCO and not just lab specific. We are starting a Sev1 with IT Networking at Seagate.

@kpricepc
Copy link

kpricepc commented Aug 26, 2020 •
edited
Loading

The issue seems to be with Github load balancers. If we use a different load balancer, we see expected results.

[509589@ssc-vm-0001 ~]$ git clone git@lb-192-30-255-113-sea.github.com:Seagate/cortx-csm.git
Cloning into 'cortx-csm'...
The authenticity of host 'lb-192-30-255-113-sea.github.com (192.30.255.113)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
RSA key fingerprint is MD5:16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lb-192-30-255-113-sea.github.com,192.30.255.113' (RSA) to the list of known hosts.
remote: Enumerating objects: 372, done.
remote: Counting objects: 100% (372/372), done.
remote: Compressing objects: 100% (285/285), done.
remote: Total 57547 (delta 160), reused 146 (delta 77), pack-reused 57175
Receiving objects: 100% (57547/57547), 42.16 MiB | 20.98 MiB/s, done.
Resolving deltas: 100% (27082/27082), done.
[509589@ssc-vm-0001 ~]$ git clone git@lb-140-82-112-4-iad.github.com:Seagate/cortx-csm.git
Cloning into 'cortx-csm'...
remote: Enumerating objects: 372, done.
remote: Counting objects: 100% (372/372), done.
remote: Compressing objects: 100% (285/285), done.
Receiving objects:  17% (9783/57547), 3.25 MiB | 38.00 KiB/s

@kpricepc
Copy link

I've submitted a ticket here with GitHub support...

https://enterprise.githubsupport.com/hc/en-us/requests/814338

I've also tried to call them and haven't been able to reach anyone.
I am still working with IT as they could possibly route traffic a different direction if needed. However this is more risky as it affects all of the LCO site.

@kpricepc
Copy link

I was able to get a hold of GitHub support on the phone and they had a similar report from another Colorado customer and they brought the IP address of their load balancer we were also having issue with out of the loop. The issue seems to be resolved from our end and I can do quick clones now. Looks like the Jenkins jobs are also able to checkout now.

This issue should be resolved so let the Lab team know otherwise.

@mukul-seagate11
Copy link
Contributor

image

Issue resolved

@gowthamchinna
Copy link
Contributor Author

@mukul-seagate11 Issue happening again

http://eos-jenkins.colo.seagate.com/job/Release_Engineering/job/github-work/job/GitHub%20Multibranch%20Jobs/job/motr-multibranch/job/PR-253/2/console

@gowthamchinna gowthamchinna reopened this Aug 31, 2020
@stale
Copy link

stale bot commented Sep 7, 2020

This issue/pull request has been automatically marked as stale because it has not had recent activity. Tagging @shailesh-vaidya

@stale stale bot added the auto-stale label Sep 7, 2020
@shailesh-vaidya
Copy link
Contributor

@gowthamchinna I feel we are not observing this issue again. Could you please confirm and if yes mark this one as closed.

@stale stale bot removed the auto-stale label Sep 7, 2020
@stale
Copy link

stale bot commented Sep 14, 2020

This issue/pull request has been automatically marked as stale because it has not had recent activity. Tagging @shailesh-vaidya

@stale stale bot added the auto-stale label Sep 14, 2020
@gowthamchinna
Copy link
Contributor Author

No issues found for last 2 weeks closing this.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@mukul-seagate11 mukul-seagate11

Labels
bug Bug in script or CI workflow build failure Build failure due to CI scripts or infrastructure issue.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@gowthamchinna @shailesh-vaidya @kpricepc @mukul-seagate11