mac [Intel-chip]: flutter commands are throwing Exhausted heap space, trying to allocate 32 bytes. ../../runtime/vm/object.cc: 2717: error: Out of memory. error
#104387
Comments
darshankawar
added
in triage
|
/cc @mraleph |
darshankawar
added
perf: memory
|
@darshankawar could you try bisecting the git history to see which commit caused this? /cc @rmacnak-google fyi |
I tried bisecting it and it seem to point to #101006 but I am not sure if that' the correct bisect. |
|
Here is the complete error |
|
I got that error, too. Perhaps the version information from the first line is helpful: My installation is completely broken now. |
|
I have checked the Workaround: if your installation is broken you can unbreak it by either checking out b8b0c80 (or earlier) or by stripping the signature via AFAIK the breakage started from ec20ea8 - but I don't think it is the roll itself that broke things. The must have been some change in the configuration that handles signing that caused this. /cc @jmagman @godofredoc @athomas for visibility |
|
Removing the signature did indeed help. |
|
I have identified the change to the internal recipes that likely caused this and notified the author. |
|
Let's revert it but the reason for the breakage may be different. https://flutter-review.googlesource.com/c/recipes/+/30202 is only adding new dependencies which run at packaging time as the last step of the release process. |
|
On 05/17 we started using dart artifacts from CIPD packages maybe something different on cipd compared to the GCS bundles? flutter/engine@051eee8 |
Let me take a look at if that is the cause... |
dnfield
added
P0
|
Marking P0. |
|
Hmm, I can't reproduce this at flutter/engine@051eee8, I'll have to bisect further |
|
Lemme see if I can reproduce: |
|
Hmm, The tool compiles fine at #101006. Which makes me think this is flaky and not consistently failing... |
|
Ok, I merged my engine revert; I will monitor post-submits and manually roll to the framework if we're all green |
|
cc @XilaiZhang |
|
I'm not sure if this fully explains it, but it looks like this passed on bots running macOS 12.0.1, but failed on bots running 12.2. I'm seeing failures on macOS 12.4. |
Manual engine roll to work around flutter#104387 d882b2d Revert "Use CIPD to download prebuilt Dart SDKs (flutter#33345)" (flutter#33558) 4bff087 Roll Skia from d3cbea4114aa to 86323276b2fc (4 revisions) (flutter#33556) 80b183e [Windows] Add textScaleFactor support (flutter#33316) 9388aa1 Roll Fuchsia Mac SDK from SmBoqSpQQ... to 1URI3xQBh... (flutter#33554) 98be801 Roll Fuchsia Linux SDK from t9KwJZBWT... to KG9Y2q6J-... (flutter#33553) 0c591bf Roll Skia from fb7f03aeac6e to d3cbea4114aa (1 revision) (flutter#33550) 945cb27 reduce the number of dependencies declared by flutter_frontend_server (flutter#33520) 76137e8 Roll Fuchsia Mac SDK from yKCDVbBgS... to SmBoqSpQQ... (flutter#33542) dc49d18 Roll Fuchsia Linux SDK from 3mdV9nX-U... to t9KwJZBWT... (flutter#33541) 6a4c45a Roll Dart SDK from d3e96bbc3d87 to 7fa990079a0a (1 revision) (flutter#33540) 5a3a0c1 Roll Skia from ade84dfd2ba5 to fb7f03aeac6e (1 revision) (flutter#33539) 4db98a0 Roll Fuchsia Mac SDK from 1QcM2qaSN... to yKCDVbBgS... (flutter#33538) f13d03c Roll Dart SDK from 708801716094 to d3e96bbc3d87 (1 revision) (flutter#33537) 2d3affd Roll Fuchsia Linux SDK from asLtG1Pbw... to 3mdV9nX-U... (flutter#33536) dfa4724 Roll Dart SDK from 90d5fdb9368b to 708801716094 (1 revision) (flutter#33534)
|
Manual engine roll of my revert out for review: #104431 |
|
It's pretty mysterious. If I download SDK from cloud storage, I get signed $ gsutil.py cp gs://flutter_infra_release/flutter/1bfe551d6b5adec64bb677cbb31ffd56c5acff49/dart-sdk-darwin-x64.zip .
$ unzip dart-sdk-darwin-x64.zip
$ otool -l dart-sdk/bin/dart | grep -A2 LC_UU
cmd LC_UUID
cmdsize 24
uuid 1E5DE27B-F121-3585-94C4-00E40C5BBF08
$ codesign -dv dart-sdk/bin/dart
Executable=/Users/vegorov/dart-sdk/bin/dart
Identifier=dart
Format=Mach-O thin (x86_64)
CodeDirectory v=20500 size=304080 flags=0x10000(runtime) hashes=9497+2 location=embedded
Signature size=8978
Timestamp=22 May 2022 at 02.08.42
Info.plist=not bound
TeamIdentifier=S8QB4VV633
Runtime Version=11.1.0
Sealed Resources=none
Internal requirements count=1 size=164If I download the binary from CIPD then I get unsigned binary (and it's the very same binary!): $ cat ensure_file
flutter/dart-sdk/mac-amd64 PAS9kiLp-LpB4gbORggYeSJMFUan4cd2Dr3ZmraJZlwC
$ cipd ensure -root sdk_root -ensure-file ensure_file
$ otool -l sdk_root/bin/dart | grep -A3 LC_UU
cmd LC_UUID
cmdsize 24
uuid 1E5DE27B-F121-3585-94C4-00E40C5BBF08
$ codesign -dv sdk_root/bin/dart
sdk_root/bin/dart: code object is not signed at allAlso when I look at other (subsequent) engine builds they seem to be fine. So we don't have to actually revert anything. I wonder if somebody run some testing script and replaced an SDK build in the bucket? |
Wow, I assumed the asset CIPD was downloading had gotten retroactively codesigned, but if CIPD still yields the unsigned binary...I'm really confused. |
Isn't cloud storage where the signed binaries live?
CIPD is content addressed storage, if something modifies the package it gets a different hash. The Dart binaries on flutter's CIPD namespace are unsigned.
|
|
Yeah, so it looks like this was signed on the Flutter side: cc @XilaiZhang |
|
@athomas These ones are not supposed to be signed IIRC, at least not yet. If I download a different build it is unsigned (e.g. dfa4724 is the next engine build and it is unsigned). So I'd say it is unexpected that this particular SDK build is signed. It's supposed to be just a copy of Dart SDK build, but it is somehow got signed with Flutter's certificates. And we can't find the place in the code where it happened. Nothing in https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20Host%20Engine/16012/overview indicates there was any signing steps. |
|
Does Flutter have cloud storage audit logs enabled? We could look at that to figure out if something overwrote the engine build by accident. |
I believe this was, as you suspected, a manual test that was run on production binaries incorrectly. And I think you're right, we didn't need a revert. |
|
I will continue to investigate to confirm how these binaries got signed. However, to unblock the Framework tree, I think I will still land #104431, since it has unsigned binaries. Once I can verify how these were getting signed and guarantee no future binaries will be codesigned, I will unroll my revert (or re-land Zach's change). I'm also working on a retrospective doc which I will share with interested parties. |
Manual engine roll to work around #104387 d882b2d Revert "Use CIPD to download prebuilt Dart SDKs (#33345)" (#33558) 4bff087 Roll Skia from d3cbea4114aa to 86323276b2fc (4 revisions) (#33556) 80b183e [Windows] Add textScaleFactor support (#33316) 9388aa1 Roll Fuchsia Mac SDK from SmBoqSpQQ... to 1URI3xQBh... (#33554) 98be801 Roll Fuchsia Linux SDK from t9KwJZBWT... to KG9Y2q6J-... (#33553) 0c591bf Roll Skia from fb7f03aeac6e to d3cbea4114aa (1 revision) (#33550) 945cb27 reduce the number of dependencies declared by flutter_frontend_server (#33520) 76137e8 Roll Fuchsia Mac SDK from yKCDVbBgS... to SmBoqSpQQ... (#33542) dc49d18 Roll Fuchsia Linux SDK from 3mdV9nX-U... to t9KwJZBWT... (#33541) 6a4c45a Roll Dart SDK from d3e96bbc3d87 to 7fa990079a0a (1 revision) (#33540) 5a3a0c1 Roll Skia from ade84dfd2ba5 to fb7f03aeac6e (1 revision) (#33539) 4db98a0 Roll Fuchsia Mac SDK from 1QcM2qaSN... to yKCDVbBgS... (#33538) f13d03c Roll Dart SDK from 708801716094 to d3e96bbc3d87 (1 revision) (#33537) 2d3affd Roll Fuchsia Linux SDK from asLtG1Pbw... to 3mdV9nX-U... (#33536) dfa4724 Roll Dart SDK from 90d5fdb9368b to 708801716094 (1 revision) (#33534)
|
We should be in a good state as of 0015ed2 |
|
@christopherfujino should we lower the priority on this now? Based on offline conversation it sounds like there's a bit of work to do around auditing the GCS bucket and fixing some potential bugs in the way entitlements were getting added to the dart binary, but that we're otherwise in the clear now? |
I can confirm that manually testing the codesign script accidentally and incorrectly codesigned production binaries. This has been addressed, so I think moving forward we are in a good state. I will close this bug now and link the follow up retrospective action item issues. |
|
Really sorry for the havoc caused, I was testing an unfinished code signing script and caused erroneous binaries to be uploaded. The 4 engine hashes I used for testing were 1bfe551, fce741d, eba6dba, 7bde72a. Those 4 binaries could cause problems since they were not correctly code signed. Thanks a ton everyone for quickly detecting the problem! I will follow #104450 and make sure bad testings never happen again. My bad, very very sincere apologies. |
darshankawar
added
the
r: fixed
|
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of |
There is a work around for this bug
Bug
Since today morning, we observed that running any
fluttercommand such asflutter upgradeorflutter channelis throwing below error only on intel-chip mac machines.Exhausted heap space, trying to allocate 32 bytes. ../../runtime/vm/object.cc: 2717: error: Out of memory.Below is complete log for
flutter upgrade --forceandflutter channel:flutter upgrade --force log
flutter channel log
This seems to be only happening on mac machine with intel-chip, as our colleagues with M1 chip confirmed that they didn't see such error while upgrading to master or while running any other flutter commands.
flutter upgrade on M1
@maheshmnj also confirmed the same works on Windows machine too properly.
flutter upgrade on windows machine
The text was updated successfully, but these errors were encountered: