Skip to content

Conversation

samestep
Copy link
Contributor

This PR extends .jenkins/pytorch/print_sccache_log.py to filter out a distracting "error" message that @walterddr came across while debugging failures in #55176:

=================== sccache compilation log ===================
ERROR 2021-04-05T15:44:18Z: sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp: In function ‘int main()’:\n/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp:2:23: error: expected ‘;’ before ‘}’ token\n int main() { return 0 }\n                       ^\n" }

Test plan:

TODO (reviewers: is there an easy way to test this?)

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Apr 16, 2021

💊 CI failures summary and remediations

As of commit 951e017 (more details on the Dr. CI page):


None of the CI failures appear to be your fault 💚



❄️ 1 failure tentatively classified as flaky

but reruns have not yet been triggered to confirm:

See CircleCI build pytorch_linux_xenial_py3_clang7_onnx_ort_test1 (1/1)

Step: "Checkout code" (full log | diagnosis details | 🔁 rerun) ❄️

fatal: the remote end hung up unexpectedly
Using SSH Config Dir '/home/circleci/.ssh'
git version 2.27.0
Cloning git repository
Cloning into '.'...
Warning: Permanently added the RSA host key for IP address '140.82.112.3' to the list of known hosts.

remote: Enumerating objects: 552841        
remote: Enumerating objects: 553774, done.        
remote: Counting objects:   0% (1/933)        
remote: Counting objects:   1% (10/933)        
remote: Counting objects:   2% (19/933)        
remote: Counting objects:   3% (28/933)        
remote: Counting objects:   4% (38/933)        
remote: Counting objects:   5% (47/933)        
remote: Counting objects:   6% (56/933)        
remote: Counting objects:   7% (66/933)        
remote: Counting objects:   8% (75/933)        
remote: Counting objects:   9% (84/933)        
remote: Counting objects:  10% (94/933)        
remote: Counting objects:  11% (103/933)        
remote: Counting objects:  12% (112/933)        
remote: Counting objects:  13% (122/933)        
remote: Counting objects:  14% (131/933)        
remote: Counting objects:  15% (140/933)        
remote: Counting objects:  16% (150/933)        
remote: Counting objects:  17% (159/933)        
remote: Counting objects:  18% (168/933)        
remote: Counting objects:  19% (178/933)        
remote: Counting objects:  20% (187/933)        
remote: Counting objects:  21% (196/933)        
remote: Counting objects:  22% (206/933)        
remote: Counting objects:  23% (215/933)        
remote: Counting objects:  24% (224/933)        
remote: Counting objects:  25% (234/933)        
remote: Counting objects:  26% (243/933)        
remote: Counting objects:  27% (252/933)        
remote: Counting objects:  28% (262/933)        
remote: Counting objects:  29% (271/933)        
remote: Counting objects:  30% (280/933)        
remote: Counting objects:  31% (290/933)        
remote: Counting objects:  32% (299/933)        
remote: Counting objects:  33% (308/933)        
remote: Counting objects:  34% (318/933)        
remote: Counting objects:  35% (327/933)        
remote: Counting objects:  36% (336/933)        
remote: Counting objects:  37% (346/933)        
remote: Counting objects:  38% (355/933)        
remote: Counting objects:  39% (364/933)        
remote: Counting objects:  40% (374/933)        
remote: Counting objects:  41% (383/933)        
remote: Counting objects:  42% (392/933)        
remote: Counting objects:  43% (402/933)        
remote: Counting objects:  44% (411/933)        
remote: Counting objects:  45% (420/933)        
remote: Counting objects:  46% (430/933)        
remote: Counting objects:  47% (439/933)        
remote: Counting objects:  48% (448/933)        
remote: Counting objects:  49% (458/933)        
remote: Counting objects:  50% (467/933)        
remote: Counting objects:  51% (476/933)        
remote: Counting objects:  52% (486/933)        
remote: Counting objects:  53% (495/933)        
remote: Counting objects:  54% (504/933)        
remote: Counting objects:  55% (514/933)        
remote: Counting objects:  56% (523/933)        
remote: Counting objects:  57% (532/933)        
remote: Counting objects:  58% (542/933)        
remote: Counting objects:  59% (551/933)        
remote: Counting objects:  60% (560/933)        
remote: Counting objects:  61% (570/933)        
remote: Counting objects:  62% (579/933)        
remote: Counting objects:  63% (588/933)        
remote: Counting objects:  64% (598/933)        
remote: Counting objects:  65% (607/933)        
remote: Counting objects:  66% (616/933)        
remote: Counting objects:  67% (626/933)        
remote: Counting objects:  68% (635/933)        
remote: Counting objects:  69% (644/933)        
remote: Counting objects:  70% (654/933)        
remote: Counting objects:  71% (663/933)        
remote: Counting objects:  72% (672/933)        
remote: Counting objects:  73% (682/933)        
remote: Counting objects:  74% (691/933)        
remote: Counting objects:  75% (700/933)        
remote: Counting objects:  76% (710/933)        
remote: Counting objects:  77% (719/933)        
remote: Counting objects:  78% (728/933)        
remote: Counting objects:  79% (738/933)        
remote: Counting objects:  80% (747/933)        
remote: Counting objects:  81% (756/933)        
remote: Counting objects:  82% (766/933)        
remote: Counting objects:  83% (775/933)        
remote: Counting objects:  84% (784/933)        
remote: Counting objects:  85% (794/933)        
remote: Counting objects:  86% (803/933)        
remote: Counting objects:  87% (812/933)        
remote: Counting objects:  88% (822/933)        
remote: Counting objects:  89% (831/933)        
remote: Counting objects:  90% (840/933)        
remote: Counting objects:  91% (850/933)        
remote: Counting objects:  92% (859/933)        
remote: Counting objects:  93% (868/933)        
remote: Counting objects:  94% (878/933)        
remote: Counting objects:  95% (887/933)        
remote: Counting objects:  96% (896/933)        
remote: Counting objects:  97% (906/933)        
remote: Counting objects:  98% (915/933)        
remote: Counting objects:  99% (924/933)        
remote: Counting objects: 100% (933/933)        
remote: Counting objects: 100% (933/933), done.        
remote: Compressing objects:   0% (1/520)        
remote: Compressing objects:   1% (6/520)        
remote: Compressing objects:   2% (11/520)        
remote: Compressing objects:   3% (16/520)        
remote: Compressing objects:   4% (21/520)        
remote: Compressing objects:   5% (26/520)        
remote: Compressing objects:   6% (32/520)        
remote: Compressing objects:   7% (37/520)        
remote: Compressing objects:   8% (42/520)        
remote: Compressing objects:   9% (47/520)        
remote: Compressing objects:  10% (52/520)        
remote: Compressing objects:  11% (58/520)        
remote: Compressing objects:  12% (63/520)        
remote: Compressing objects:  13% (68/520)        
remote: Compressing objects:  14% (73/520)        
remote: Compressing objects:  15% (78/520)        
remote: Compressing objects:  16% (84/520)        
remote: Compressing objects:  17% (89/520)        
remote: Compressing objects:  18% (94/520)        
remote: Compressing objects:  19% (99/520)        
remote: Compressing objects:  20% (104/520)        
remote: Compressing objects:  21% (110/520)        
remote: Compressing objects:  22% (115/520)        
remote: Compressing objects:  23% (120/520)        
remote: Compressing objects:  24% (125/520)        
remote: Compressing objects:  25% (130/520)        
remote: Compressing objects:  26% (136/520)        
remote: Compressing objects:  27% (141/520)        
remote: Compressing objects:  28% (146/520)        
remote: Compressing objects:  29% (151/520)        
remote: Compressing objects:  30% (156/520)        
remote: Compressing objects:  31% (162/520)        
remote: Compressing objects:  32% (167/520)        
remote: Compressing objects:  33% (172/520)        
remote: Compressing objects:  34% (177/520)        
remote: Compressing objects:  35% (182/520)        
remote: Compressing objects:  36% (188/520)        
remote: Compressing objects:  37% (193/520)        
remote: Compressing objects:  38% (198/520)        
remote: Compressing objects:  39% (203/520)        
remote: Compressing objects:  40% (208/520)        
remote: Compressing objects:  41% (214/520)        
remote: Compressing objects:  42% (219/520)        
remote: Compressing objects:  43% (224/520)        
remote: Compressing objects:  44% (229/520)        
remote: Compressing objects:  45% (234/520)        
remote: Compressing objects:  46% (240/520)        
remote: Compressing objects:  47% (245/520)        
remote: Compressing objects:  48% (250/520)        
remote: Compressing objects:  49% (255/520)        
remote: Compressing objects:  50% (260/520)        
remote: Compressing objects:  51% (266/520)        
remote: Compressing objects:  52% (271/520)        
remote: Compressing objects:  53% (276/520)        
remote: Compressing objects:  54% (281/520)        
remote: Compressing objects:  55% (286/520)        
remote: Compressing objects:  56% (292/520)        
remote: Compressing objects:  57% (297/520)        
remote: Compressing objects:  58% (302/520)        
remote: Compressing objects:  59% (307/520)        
remote: Compressing objects:  60% (312/520)        
remote: Compressing objects:  61% (318/520)        
remote: Compressing objects:  62% (323/520)        
remote: Compressing objects:  63% (328/520)        
remote: Compressing objects:  64% (333/520)        
remote: Compressing objects:  65% (338/520)        
remote: Compressing objects:  66% (344/520)        
remote: Compressing objects:  67% (349/520)        
remote: Compressing objects:  68% (354/520)        
remote: Compressing objects:  69% (359/520)        
remote: Compressing objects:  70% (364/520)        
remote: Compressing objects:  71% (370/520)        
remote: Compressing objects:  72% (375/520)        
remote: Compressing objects:  73% (380/520)        
remote: Compressing objects:  74% (385/520)        
remote: Compressing objects:  75% (390/520)        
remote: Compressing objects:  76% (396/520)        
remote: Compressing objects:  77% (401/520)        
remote: Compressing objects:  78% (406/520)        
remote: Compressing objects:  79% (411/520)        
remote: Compressing objects:  80% (416/520)        
remote: Compressing objects:  81% (422/520)        
remote: Compressing objects:  82% (427/520)        
remote: Compressing objects:  83% (432/520)        
remote: Compressing objects:  84% (437/520)        
remote: Compressing objects:  85% (442/520)        
remote: Compressing objects:  86% (448/520)        
remote: Compressing objects:  87% (453/520)        
remote: Compressing objects:  88% (458/520)        
remote: Compressing objects:  89% (463/520)        
remote: Compressing objects:  90% (468/520)        
remote: Compressing objects:  91% (474/520)        
remote: Compressing objects:  92% (479/520)        
remote: Compressing objects:  93% (484/520)        
remote: Compressing objects:  94% (489/520)        
remote: Compressing objects:  95% (494/520)        
remote: Compressing objects:  
Receiving objects:   0% (1/553774)
Receiving objects:   1% (5538/553774)
Receiving objects:   2% (11076/553774)
Receiving objects:   3% (16614/553774)
Receiving objects:   4% (22151/553774)
Receiving objects:   5% (27689/553774)
Receiving objects:   6% (33227/553774), 8.97 MiB | 17.92 MiB/s
Receiving objects:   7% (38765/553774), 8.97 MiB | 17.92 MiB/s
Receiving objects:   8% (44302/553774), 8.97 MiB | 17.92 MiB/s
Receiving objects:   9% (49840/553774), 8.97 MiB | 17.92 MiB/s
Receiving objects:   9% (54063/553774), 8.97 MiB | 17.92 MiB/s
Receiving objects:  10% (55378/553774), 18.05 MiB | 18.04 MiB/s
Receiving objects:  11% (60916/553774), 18.05 MiB | 18.04 MiB/s
Receiving objects:  12% (66453/553774), 18.05 MiB | 18.04 MiB/s
Receiving objects:  13% (71991/553774), 18.05 MiB | 18.04 MiB/s
Receiving objects:  14% (77529/553774), 27.11 MiB | 18.07 MiB/s
Receiving objects:  15% (83067/553774), 27.11 MiB | 18.07 MiB/s
Receiving objects:  15% (83443/553774), 27.11 MiB | 18.07 MiB/s
Receiving objects:  16% (88604/553774), 37.45 MiB | 18.72 MiB/s
Receiving objects:  17% (94142/553774), 37.45 MiB | 18.72 MiB/s
Receiving objects:  18% (99680/553774), 48.30 MiB | 19.32 MiB/s
Receiving objects:  19% (105218/553774), 48.30 MiB | 19.32 MiB/s
Receiving objects:  20% (110755/553774), 48.30 MiB | 19.32 MiB/s
Receiving objects:  20% (114504/553774), 48.30 MiB | 19.32 MiB/s
Receiving objects:  21% (116293/553774), 62.26 MiB | 20.75 MiB/s
Receiving objects:  22% (121831/553774), 62.26 MiB | 20.75 MiB/s
Receiving objects:  23% (127369/553774), 62.26 MiB | 20.75 MiB/s
Receiving objects:  24% (132906/553774), 76.45 MiB | 21.84 MiB/s
Receiving objects:  25% (138444/553774), 76.45 MiB | 21.84 MiB/s
Receiving objects:  25% (143457/553774), 76.45 MiB | 21.84 MiB/s
Receiving objects:  26% (143982/553774), 90.26 MiB | 22.56 MiB/s
Receiving objects:  27% (149519/553774), 90.26 MiB | 22.56 MiB/s
Receiving objects:  28% (155057/553774), 90.26 MiB | 22.56 MiB/s
Receiving objects:  29% (160595/553774), 102.97 MiB | 22.89 MiB/s
Receiving objects:  30% (166133/553774), 102.97 MiB | 22.89 MiB/s
Receiving objects:  31% (171670/553774), 102.97 MiB | 22.89 MiB/s
Receiving objects:  31% (173855/553774), 102.97 MiB | 22.89 MiB/s
Receiving objects:  32% (177208/553774), 115.78 MiB | 23.74 MiB/s
Receiving objects:  33% (182746/553774), 115.78 MiB | 23.74 MiB/s
Receiving objects:  34% (188284/553774), 129.21 MiB | 24.71 MiB/s
Receiving objects:  35% (193821/553774), 129.21 MiB | 24.71 MiB/s
Receiving objects:  35% (196463/553774), 144.43 MiB | 26.07 MiB/s
Receiving objects:  35% (196463/553774), 184.18 MiB | 30.20 MiB/s
ssh_dispatch_run_fatal: Connection to 140.82.112.3 port 22: message authentication code incorrect

fatal: the remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed


exit status 128

CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
/bin/bash: line 19: unbuffer: command not found
/bin/bash: line 19: ts: command not found


This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

@facebook-github-bot
Copy link
Contributor

@samestep has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Copy link
Contributor

@driazati driazati left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like any time sccache appears in the logs it's basically useless to devs (especially when Dr. CI picks it up), should we skip outputting its logs altogether (or somehow shove them somewhere where you have to go looking for them to get it)?

Copy link
Contributor

@walterddr walterddr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good.

RE: Test plan: since we only added one string to the list, it should be fine

@walterddr
Copy link
Contributor

Seems like any time sccache appears in the logs it's basically useless to devs (especially when Dr. CI picks it up), should we skip outputting its logs altogether (or somehow shove them somewhere where you have to go looking for them to get it)?

It is useful to debug whether there's any issue regarding build time regression. no necessarily for a failure though

Co-authored-by: Rong Rong <walterddr.walterddr@gmail.com>
@facebook-github-bot
Copy link
Contributor

@samestep has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@samestep merged this pull request in 7629477.

krshrimali pushed a commit to krshrimali/pytorch that referenced this pull request May 19, 2021
Summary:
This PR extends `.jenkins/pytorch/print_sccache_log.py` to filter out a distracting "error" message that walterddr came across while debugging failures in pytorch#55176:

```
=================== sccache compilation log ===================
ERROR 2021-04-05T15:44:18Z: sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp: In function ‘int main()’:\n/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp:2:23: error: expected ‘;’ before ‘}’ token\n int main() { return 0 }\n                       ^\n" }
```

Pull Request resolved: pytorch#56281

Test Plan: TODO (reviewers: is there an easy way to test this?)

Reviewed By: walterddr

Differential Revision: D27826064

Pulled By: samestep

fbshipit-source-id: 7322a830c1246820a5b2b7bbeaa4697ebd13b617
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants