New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Multiboot buffer is too small." after upgrade to ESXi-6.7.0-20181002001-standard (Build 10302608) #1

Open
giro83 opened this Issue Oct 21, 2018 · 43 comments

Comments

Projects
None yet
@giro83

giro83 commented Oct 21, 2018

Hi,

Apologies if this is the wrong place where to report my issue. Feel free to say so should this be the case.

I have a 2014 Apple Mac Mini (the last 2.6GHz Quad-Core Intel Core i7 they made) on which I run ESXi (for home/educational purposes only - no production stuff on it, clearly).

I've run several ESXi versions on it over the years: 5.5, 6.0, 6.5 and now 6.7. It's the perfect device on which to run ESXi at home (except for the price), given it's small, quiet, stylish, etc.

Anyway, today I upgraded from ESXi-6.7.0-20181004001-standard (Build 10176752) to ESXi-6.7.0-20181002001-standard (Build 10302608). The upgrade has essentially "bricked" the device.

ESXi is seen booting and loading files, but after a few seconds returns:

Shutting down firmware services...
Multiboot buffer is too small.
Unrecoverable error

Also attaching a photo of the screen below.

I searched around, and I couldn't find references to this error on the Internet. However, I have found this project, and it seems the error comes from here:

Log(LOG_ERR, "Mutiboot buffer is too small.\n");

Again, I'm fully aware my host is not a supported device... but it does the job for me, and I have no reason to replace it yet.

Can I do anything to recover it, short of performing a factory reset of 6.7.0 and not upgrading to this imageprofile or newer?

Is this error related to UEFI boot?

Can I switch to legacy boot?

Can I pass any parameter to the kernel line to increase the buffer which the error refers to?

Thanks in advance,
Stefano

img_0027

@giro83

This comment has been minimized.

giro83 commented Oct 22, 2018

Screen grab when booting with Shift+V:

img_0047

@andreiw

This comment has been minimized.

andreiw commented Oct 23, 2018

Hi @giro83, thanks for the report. There's one thing we could try, but need to get access to a UEFI shell. Unfortunately, this is a Mac we're talking about - you wouldn't happen to have rEFInd or rEFIt installed, would you?

@giro83

This comment has been minimized.

giro83 commented Oct 23, 2018

Hi @andreiw, I can boot the Mac Mini with a (e.g.) Gentoo Live USB stick. I can get networking going and install packages needed (probably including the ones you mentioned). I should then be able to execute commands / follow a procedure.

https://wiki.gentoo.org/wiki/Refind

@andreiw

This comment has been minimized.

andreiw commented Oct 23, 2018

Okay, here's how I could get boot to the UEFI shell on a mac.

This should work on any OS, but I will use OS X here.

  1. Download http://downloads.sourceforge.net/refit/refit-bin-0.14.tar.gz
  2. Uncompress to a FAT32 formatted USB key
  3. Rename "\efi\refit" to "\efi\boot"
  4. Rename "\efi\boot\refit.efi" to "\efi\boot\bootx64.efi"

Reboot, hold "Option key", select "UEFI shell" (it has a terminal icon).

Once you have the Shell running, find the \efi\boot\bootx64.efi file corresponding to the ESX installation. Use "map" command to print the available UEFI equivalents of DOS "drive letters", and
"cd" and "dir" to navigate and list. E.g.:

Shell> fs5:
Shell> cd efi\boot\
Shell> dir

Once you find it, run it with the "-U" option:

Shell> bootx64.efi -U 
@andreiw

This comment has been minimized.

andreiw commented Oct 23, 2018

Hint: there will be a lot of output inside "map". To pause while scrolling run it like this, with the "-b" argument:

 Shell> map -r -b
@andreiw

This comment has been minimized.

andreiw commented Oct 23, 2018

Just a visual example of what the USB stick directory structure looks like, compared to the refit TAR file:

Top is USB stick. Bottom is tar file.

image

@quotesbnbi

This comment has been minimized.

quotesbnbi commented Oct 23, 2018

I am having the same error pictures on a MacPro6,1 after attempting to upgrade to the listed version. I reboot after the vib says it installed correctly and that is the error I receive. If I power cycle the mac, it boots fine and is back at build 10176752. Unfortunately I cannot try the steps advised due to restrictions. All the same, know that it affects not just the Mac Mini, but the Pro(trash can) as well.

I have a due date of getting security issues that the new patch mitigate of early Dec. If it a solution isn't available by then I'll be forced to take the system offline which will be a major headache for me because I rely on it heavily for testing of our management policies, patches, scripts, etc. Fingers crossed!

If there is any information i can pull using the esxi built in tools that would be helpful, please let me know.

@giro83

This comment has been minimized.

giro83 commented Oct 23, 2018

@andreiw Success! You have totally resurrected my ESXi installation :)

If I understood correctly what I just did, this seems to have (permanently?) disabled UEFI. I have verified that the host can now boot ESXi (as before) completely unattended.

Do I need to worry about future updates re-enabling this for me?

Will you be doing anything at your end?

Anyway, thank you very much so far.

@giro83

This comment has been minimized.

giro83 commented Oct 23, 2018

Mine also seems to be back at build 10176752, however. Should I attempt the upgrade again?

@andreiw

This comment has been minimized.

andreiw commented Oct 23, 2018

Hi @giro83. Thanks for doing the steps to help triage the issue. This option boots without further EFI support in ESX (for the current boot, not permanently). We'll need to investigate this further internally to diagnose the actual cause.

I do want to point out that this Github issue tracker is not the right channel to raise customer support issues or get support. Our communication here, as useful as it is to get data, does not constitute any acknowledgement/receipt of a support issue, or provide a guarantee/promise of remediation. Please raise a support request via https://www.vmware.com/support/file-sr.html, and you should contact your TAM if your have one. Filing an SR is the way forward to report and address product issues and defects (e.g. @quotesbnbi).

Thanks for your help in identifying the symptoms and the affected systems, and thanks for the heads up. I'll close this as there is nothing else to be done via this (unofficial) channel.

@andreiw andreiw closed this Oct 23, 2018

@andreiw

This comment has been minimized.

andreiw commented Oct 24, 2018

@giro83 and @quotesbnbi, would it be possible to get more system configuration information on the machines you've seen the problem with?

  • Make/year/model of machine. E.g. MacBook Pro (Retina, 15-inch, Mid 2015), MacBookPro11,5.
  • How much RAM the system has.
  • Boot ROM and SMC versions, if possible (e.g. MBP114.0172.B25 and 2.30f2).

@andreiw andreiw reopened this Oct 24, 2018

@NM-IT

This comment has been minimized.

NM-IT commented Oct 24, 2018

Just so you know you're not alone: We also got stuck with this problem this weekend while (trying to) upgrade to 6.7U1 from 6.7.0, 10176752. We also tried a complete reinstall from VMware-VMvisor-Installer-6.7.0.update01-10302608.x86_64.iso - but after reboot got stuck just the same.
We run 3 x MacPro6.1 hosts (Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz) with 128GB RAM (from OWC).

@giro83

This comment has been minimized.

giro83 commented Oct 24, 2018

Just so you know you're not alone: We also got stuck with this problem this weekend while (trying to) upgrade to 6.7U1 from 6.7.0, 10176752. We also tried a complete reinstall from VMware-VMvisor-Installer-6.7.0.update01-10302608.x86_64.iso - but after reboot got stuck just the same.
We run 3 x MacPro6.1 hosts (Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz) with 128GB RAM (from OWC).

This is interesting/useful information, because although my host is unsupported, the Mac Pro is supported according to https://www.vmware.com/resources/compatibility/search.php

@NM-IT

This comment has been minimized.

NM-IT commented Oct 24, 2018

Yes that hardware is supported - so this really bugs me. Unfortunately my machines are in production (offsite@colocation center) - so I can't really dig into troubleshooting this more at this point.
But I did open a SR just now, so I hope VMware will investigate this and possibly fix it.

@giro83

This comment has been minimized.

giro83 commented Oct 24, 2018

But @andreiw just to be clear, I totally appreciate what you've done for me (outside of any official support channel). It's more than I expected when I opened this issue, and I'm really happy.

I will try to figure out on my own why my ESXi installation is now booting correctly unattended (albeit from the old version). Maybe there was more than one ESXi installation to boot from and I picked the wrong one, I don't know.

I will also supply the info you requested ASAP.

@quotesbnbi

This comment has been minimized.

quotesbnbi commented Oct 24, 2018

We run the following hardware:
MacPro6,1, Late 2013, A1481
32GB memory
12 core processor: 12 CPUs x Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz
Bios: MP61.88Z.0120.B00.1708080652
Bios release date: Monday, August 07, 2017, 20:00:00 -0400

@lamw

This comment has been minimized.

lamw commented Oct 24, 2018

Disclaimer: VMware Employee

Thanks @andreiw for forwarding this internally. We're trying to see if we can reproduce this problem with Engr. We definitely like to get as much information as possible (configuration and how the USB Installer was created) to make sure we're running the exact same workflow.

@NM-IT The Mac Pro 6,1 is indeed a supported platform and thanks for filing an SR (this is the best way to engage with VMware, so our support team can work with you). Would you mind sharing the SR number?

@quotesbnbi Thanks for the info, I'll share this back with the Engr team

@NM-IT

This comment has been minimized.

NM-IT commented Oct 24, 2018

VMware Support Request 18963226910 is the case number used here.

@dminion909

This comment has been minimized.

dminion909 commented Oct 24, 2018

Hi everyone. We are also having this issue. Not just with the 6.7 u1 but with all previous versions as well. We are running ESXi on quite a few MacPro 6,1 systems. This system in particular has a newer firmware version which we believe is the culprit. This is the first version of ESXi I've actually managed to get installed on this system. Every other version has had the "nfs41client failed to load" followed by no network adapters issue as previously experienced by @lamw. I am hesitant to try to flash the ROM back to a previous version as turning this system into a brick would be a pretty costly mistake. If you guys have any ideas on how to get 6.7 or 6.0 to install on this MP6,1, I'd really appreciate it.

@lamw

This comment has been minimized.

lamw commented Oct 24, 2018

@dminion909 Can you share the version of the firmware that you're seeing for the newer system? And a few of firmware version for systems without issues (unless they're all the same). Thanks

@dminion909

This comment has been minimized.

dminion909 commented Oct 24, 2018

I'm not sure of the firmware on the ones in production but they are most likely the original firmware that was installed when the units were shipped. ESX was immediately installed on them so they never had a chance to update.

This one had El Capitan installed and has updated. I managed to get ESXi 6.7 u1 HPE installed. The regular 6.7 u1 wouldn't install as it would get the Multiboot error on install. I started this system up again after getting ESXi installed and on second boot, I'm getting:
VMware Hypervisor Recovery
BANK5: invalid configuration.
BANK6: invalid configuration.
Ho hypervisor found.

I can boot from Mac OS X recovery and get the firmware of this one. It will take me a bit so I'll get it for you ASAP. Thanks @lamw

@dminion909

This comment has been minimized.

dminion909 commented Oct 24, 2018

Here you go @lamw. This is the system I'm not able to get working:

Model Name: Mac Pro
Model Identifier: MacPro6,1
Processor Name: 8-Core Intel Xeon E5
Processor Speed: 3 GHz
Number of Processors: 1
Total Number of Cores: 8
L2 Cache (per core): 256 KB
L3 Cache: 25 MB
Memory: 64 GB
Boot ROM Version: MP61.0116.B17
SMC Version (system): 2.20f18
Illumination Version: 1.4a6

@dminion909

This comment has been minimized.

dminion909 commented Oct 24, 2018

Update:

I was able to successfully install HPE Customized Image ESXi 6.7.0 Update 1 version 670.U1.10.3.5 released on October 2018 and based on ESXi 6.7.0 Update 1 Vmkernel Release Build 10302608
Installing to the local disk would not work but installing to a 16 GB flash drive worked like a charm... so far. Go figure.
6.7 is not actually the version we need though. We need version 6.0 so I'm going to try that now.

Update:

Still no dice with 6.0 u3 (9313334). I got the "nfs41client failed to load" followed by no network adapters. So I guess it's good that I'm able to install some flavor of ESX but I guess we'll have to upgrade our entire cluster now lol. So if anyone has any info, please let me know!

@jjas0n

This comment has been minimized.

jjas0n commented Oct 25, 2018

Chiming in - Have a few Late 2012 Mac Minis. All are exhibiting the issue here. rEFIt got me back online for now. I'd love to get to the bottom of thus.

Thank you!

@dminion909

This comment has been minimized.

dminion909 commented Oct 25, 2018

@jjas0n could you provide some info on what you did to get it working using rEFIt? Perhaps I could use it to fix the issue I'm running into with installing 6.0...
Thanks!

@jjas0n

This comment has been minimized.

jjas0n commented Oct 25, 2018

@dminion909 - I just followed the instructions from @andreiw above to boot from the VMWARE bootx64.efi loader with the -U flag. That’s it.

I cannot apply the 6.7.0U1 update at this point, but at least I’m running.

Hope this helps.

@dminion909

This comment has been minimized.

dminion909 commented Oct 26, 2018

Morning all. I was able to get ESXi 6.0.0 4192238 installed to the local drive on MacPro6,1. The way I was able to do this was by booting from an El Capitan USB installer, launching the Disk Utility and wiping all the partitions on the system disk. Once I did that, I booted from the ESXi USB installer and the installation ran just fine. No nfs41client issues. No issues finding the NICs. No Multiboot errors. It successfully pulled an IP and I can log in through vSphere. Haven't migrated VMs or joined the cluster yet but this is as far as I've gotten. This might fix your issues for 6.7 as well.

We also have a MacMini7,1 that wouldn't install and this worked on that system as well.

Specs:
Model Name: Mac Pro
Model Identifier: MacPro6,1
Processor Name: 8-Core Intel Xeon E5
Processor Speed: 3 GHz
Number of Processors: 1
Total Number of Cores: 8
L2 Cache (per core): 256 KB
L3 Cache: 25 MB
Memory: 64 GB
Boot ROM Version: MP61.0116.B17
SMC Version (system): 2.20f18
Illumination Version: 1.4a6

Hope that helps. Have a great weekend everyone 😄
@lamw @andreiw @jjas0n @giro83

@mightyfree

This comment has been minimized.

mightyfree commented Oct 28, 2018

I experienced this issue on a 2012 Mac Mini 6,2 hosted at Mac Stadium, upgrading from 6.7 10176752 to 6.7U1, which was unrecoverable. Remediation included installing ESXi 6.5 over existing 6.7 install using 6.5 ESXi installation media, while ensuring the option to preserve existing VMs was checked. Remote hands were needed to make this happen. The Mac mini was subsequently upgraded up to 6.7 10176879 successfully, while the 6.7U1 update is investigated and fixed.

@mightyfree

This comment has been minimized.

mightyfree commented Oct 28, 2018

@dminion909 Mac Stadium (macstadium.com) hosts hundreds of Mac Pros, many with a variety of flavors of ESXi installed. You may try reaching out to their tech support team for ideas.

@dminion909

This comment has been minimized.

dminion909 commented Oct 29, 2018

Thanks @mightyfree . I actually did early last week and they haven't had this particular issue. I'm glad you reminded me though... I'll email them back with an update.

@jorge-dbx

This comment has been minimized.

jorge-dbx commented Oct 31, 2018

Experiencing this issue as well on about a dozen of Mac Pros (the earliest ones we purchased for our fleet).
Mac Pro Boot Rom: MP61.0116.B16 (with issues)

Fixed it by installing MacOS High Sierra and connecting it to the network to download all the latest firmware updates.

Post-install Mac Pro Boot rom: MP61.0120.B00

Open case with VMware: 18970432210

@lamw hope this helps

@lamw

This comment has been minimized.

lamw commented Oct 31, 2018

Folks,

Thanks for everyones patience and support regarding this issue. We believe we have a fix in place for this problem and we've confirmed both internally and with a customer. The fix has been checked-in and will automatically be included in a future update or release of ESXi. Unfortunately I can not comment on specific timelines and/or the specific release that will include the fix.

In the mean time, I'm trying to work with Product Management to publish a KB regarding this known issue and you can subscribe to it for future updates including when the fix is available. For now, I would recommend folks holding off on upgrading to vSphere 6.7 Update 1 on Apple Mac Pro 6,1 (which is the only officially supported platform for ESXi) as well as the Mac Mini and any other non-supported Apple hardware

@NM-IT

This comment has been minimized.

NM-IT commented Nov 3, 2018

thanx @lamw for your update and support!

@s0ftice

This comment has been minimized.

s0ftice commented Nov 9, 2018

Today's ESXi-6.7.0-20181104001-standard (Build 10764712) release did not solve it for me unfortunately...

@erpomik

This comment has been minimized.

erpomik commented Nov 12, 2018

FYI: I tried to upgrade/re-install two Mac Pro's this weekend.
Got "Multiboot buffer is too small" on both.

@lamw

This comment has been minimized.

lamw commented Nov 16, 2018

FYI - https://www.virtuallyghetto.com/2018/11/heads-up-apple-mac-pro-61-fails-to-boot-after-upgrading-to-esxi-6-7-update-1.html <-Includes VMware KB, which folks can follow to hopefully help them revert back to previous ESXi installation. You can also subscribe to the KB for updates regarding this issue

@mightyfree

This comment has been minimized.

mightyfree commented Nov 18, 2018

As @jorge-dbx stated above, it appears the issue may be due to outdated boot roms on our Mac hardware. When MacOS is installed, these updates are installed as MacOS is regularly updated.

But because our hardware has ESXi installed - sometimes for years at a time, we're missing out on these important boot rom updates.

While it's possible for VMware to code a workaround, perhaps the fix is as easy as updating the boot roms per @jorge-dbx 's instructions. I'll look into this potential resolution on a 2012 Mac mini that is experiencing this error, and report back.

@quotesbnbi

This comment has been minimized.

quotesbnbi commented Nov 20, 2018

As noted earlier, I am also running the same bios as @jorge-dbx and am unable to run 6.7u1. I wiped the mac pro and installed every OS up to and including high sierra in Nov 2017 in order to get all firmware updates. Unless a newer update has come out since then that isn't being reported by @jorge-dbx, then something else is going on here. In any case, it sounds like the esxi-boot code has been fixed and we are just awaiting a release. Thought I should mention it before people start blowing away their esx data to install firmware that isn't certain to fix the issue.

Jorge's 'boot rom': MP61.0120.B00
Mine as reported by the vmware host web interface: Bios: MP61.88Z.0120.B00.1708080652

@NM-IT

This comment has been minimized.

NM-IT commented Nov 21, 2018

WMware have now responded to my support request 18963226910 (regarding "Shutting down firmware services... Multiboot buffer is too small. Unrecoverable error".
They say: "Kindly note that the issue was resolved in a new release. You can now successfully upgrade to vSphere 6.7 U2."
I haven't tried it yet - and I haven't even found the update... Anyone see it?

@shantur

This comment has been minimized.

shantur commented Nov 21, 2018

@NM-IT : Ask the build number.
Then check the build patch here : https://my.vmware.com/group/vmware/patch#search

@TimothyPMann

This comment has been minimized.

Contributor

TimothyPMann commented Nov 22, 2018

@shantur

This comment has been minimized.

shantur commented Nov 22, 2018

Maybe a patch fix then

@LordAttilas

This comment has been minimized.

LordAttilas commented Nov 26, 2018

Okay, here's how I could get boot to the UEFI shell on a mac.

This should work on any OS, but I will use OS X here.

  1. Download http://downloads.sourceforge.net/refit/refit-bin-0.14.tar.gz
  2. Uncompress to a FAT32 formatted USB key
  3. Rename "\efi\refit" to "\efi\boot"
  4. Rename "\efi\boot\refit.efi" to "\efi\boot\bootx64.efi"

Reboot, hold "Option key", select "UEFI shell" (it has a terminal icon).

Once you have the Shell running, find the \efi\boot\bootx64.efi file corresponding to the ESX installation. Use "map" command to print the available UEFI equivalents of DOS "drive letters", and
"cd" and "dir" to navigate and list. E.g.:

Shell> fs5:
Shell> cd efi\boot\
Shell> dir

Once you find it, run it with the "-U" option:

Shell> bootx64.efi -U 

That worked for me for a MacMini 5,2. Big thanks :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment