Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OVMF 64MMIO / "above 4G decoding" for Tesla pass through #59

Closed
ThomasLamprecht opened this issue Feb 22, 2016 · 9 comments
Closed

OVMF 64MMIO / "above 4G decoding" for Tesla pass through #59

ThomasLamprecht opened this issue Feb 22, 2016 · 9 comments

Comments

@ThomasLamprecht
Copy link
Contributor

Hi,

what is the state when I'd try to "PCI - pass-trough" a powerful graphic card (like the nvidia tesla K80)
to qemu with OVMF as UEFI/BIOS?

On a physical machine I may have to enable 64-bit MMIO (or like it's often called "above 4G decoding"), how is that with OVMF, as far as I have seen there is no such option in OVMF also I'm currently not sure what the state in qemu itself is I only found an old mailing post [1] regarding this issue and posted a week ago on there user/discuss list but received sadly no response yet.

So I'm hoping someone could point me in the right direction.
If this is the wrong place to ask I'm glad to move the question to an more appropriate one, but I did not found any mailing/discuss list for OVMF.

[1] https://lists.nongnu.org/archive/html/qemu-devel/2010-02/msg01106.html
@lersek lersek added the OVMF label Feb 22, 2016
@lersek
Copy link
Member

lersek commented Feb 22, 2016

Hello @GamerSource, this is the right place -- or one right place anyway -- to ask.

QEMU definitely supports 64-bit MMIO for the x86_64 target.

The PCI host bridge / root bridge driver that OVMF uses at the moment lacks support for 64-bit PCI MMIO however. This is a known limitation, and it should be resolved after OVMF is ported to the new, central PCI host bridge / root bridge driver in edk2. (It won't happen automatically with the port, but when the port is complete, it shouldn't take much work on top.)

I started to work on that port recently, but ran into some issues with the central driver that OVMF should use. We've been discussing those topics recently on the mailing list, with @marcel-apf and @niruiyu .

Please refer to these (sub)threads in the list archive:

@ThomasLamprecht
Copy link
Contributor Author

Hello @lersek thanks for the response, really appreciate it!

Ok cool, good to know, I guessed it had the capabilities but did not found that much info on it and I'm not (yet) that sound on skimming through qemu code to see what's there.

Thanks for the write up and the thread links, I guess I found a new interesting mailing list to subscribe.

As this is then really an "issue" with OVMF I feel that its probably correct to not close this here on github yet, for sake of documenting and helping maybe someone other to find out the cause why it's not working yet, feel free to do otherwise.

Thanks again!

@lersek
Copy link
Member

lersek commented Feb 23, 2016

This is a genuine, although likely not too urgent, shortcoming in OVMF, at the moment, so we should certainly keep this item open.

@lersek lersek added the Feature label Mar 3, 2016
@lersek lersek mentioned this issue Mar 9, 2016
Closed
lersek added a commit to lersek/edk2 that referenced this issue Mar 10, 2016
@lersek
OvmfPkg: PlatformPei: factor out GetFirstNonAddress()
f9fef2d 
Factor out the expression that is currently the basis of the address width
calculation into a standalone function. In the next patches we'll raise
the return value under certain circumstances.

Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Marcel Apfelbaum <marcel@redhat.com>
Cc: Thomas Lamprecht <t.lamprecht@proxmox.com>
Ref: tianocore#59
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
lersek added a commit to lersek/edk2 that referenced this issue Mar 10, 2016
@lersek
OvmfPkg: PlatformPei: determine the 64-bit PCI host aperture for X64 DXE
e760cb5 
The main observation about the 64-bit PCI host aperture is that it is the
highest part of the useful address space. It impacts the top of the GCD
memory space map, and, consequently, our maximum address width calculation
for the CPU HOB too.

Thus, modify the GetFirstNonAddress() function to consider the following
areas above the high RAM, while calculating the first non-address (i.e.,
the highest inclusive address, plus one):

- the memory hotplug area (optional, the size comes from QEMU),

- the 64-bit PCI host aperture (we set a default size).

While computing the first non-address, capture the base and the size of
the 64-bit PCI host aperture at once in PCDs, since they are natural parts
of the calculation.

(Similarly to how PcdPciMmio32* are not rewritten on the S3 resume path
(see the InitializePlatform() -> MemMapInitialization() condition), nor
are PcdPciMmio64*. Only the core PciHostBridgeDxe driver consumes them,
through our PciHostBridgeLib instance.)

Set 32GB as the default size for the aperture. Issue#59 mentions the
NVIDIA Tesla K80 as an assignable device. According to nvidia.com, these
cards may have 24GB of memory (probably 16GB + 8GB BARs).

As a strictly experimental feature, the user can specify the size of the
aperture (in MB) as well, with the QEMU option

  -fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=65536

The "X-" prefix follows the QEMU tradition (spelled "x-" there), meaning
that the property is experimental, unstable, and might go away any time.
Gerd has proposed heuristics for sizing the aperture automatically (based
on 1GB page support and PCPU address width), but such should be delayed to
a later patch (which may very well back out "X-PciMmio64Mb" then).

For "everyday" guests, the 32GB default for the aperture size shouldn't
impact the PEI memory demand (the size of the page tables that the DXE IPL
PEIM builds). Namely, we've never reported narrower than 36-bit addresses;
the DXE IPL PEIM has always built page tables for 64GB at least.

For the aperture to bump the address width above 36 bits, either the guest
must have quite a bit of memory itself (in which case the additional PEI
memory demand shouldn't matter), or the user must specify a large aperture
manually with "X-PciMmio64Mb" (and then he or she is also responsible for
giving enough RAM to the VM, to satisfy the PEI memory demand).

Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Marcel Apfelbaum <marcel@redhat.com>
Cc: Thomas Lamprecht <t.lamprecht@proxmox.com>
Ref: tianocore#59
Ref: http://www.nvidia.com/object/tesla-servers.html
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
lersek added a commit to lersek/edk2 that referenced this issue Mar 10, 2016
@lersek
OvmfPkg: PciHostBridgeLib: install 64-bit PCI host aperture
7c8bfed 
On the normal boot path (which is when PciHostBridgeDxe runs), the PCDs
have been calculated; report the 64-bit PCI host aperture to
PciHostBridgeDxe.

In the Ia32 build, the PCD values (zeros) come directly from the DEC file,
and this patch makes no difference.

Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Marcel Apfelbaum <marcel@redhat.com>
Cc: Thomas Lamprecht <t.lamprecht@proxmox.com>
Ref: tianocore#59
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
@lersek
Copy link
Member

lersek commented Mar 10, 2016

Posted series

http://thread.gmane.org/gmane.comp.bios.edk2.devel/9030

@lersek
Copy link
Member

lersek commented Mar 10, 2016

@GamerSource: I CC'd you on the patches and the cover letter (googling a combination of your name and your employer's name, taken from your github profile page, for your email). In case the emails don't reach you for any reason, I'm asking you here too to please test the patches with your hardware. Thanks!

@lersek
Copy link
Member

lersek commented Mar 14, 2016

The patch series for this issue is now "wave 2" in a larger work:

http://thread.gmane.org/gmane.comp.bios.edk2.devel/9150

@ThomasLamprecht
Copy link
Contributor Author

Thanks! I'm following your posts on the edk2 mailing list closely. I'll rebuild and send it out to test, we have the K80 currently not available so I'm dependent of an other tester...

And sorry for not providing my email address, nice detective work, though! :)
Thanks for the work!

lersek added a commit that referenced this issue Mar 23, 2016
@lersek
OvmfPkg: PlatformPei: factor out GetFirstNonAddress()
d537168 
Factor out the expression that is currently the basis of the address width
calculation into a standalone function. In the next patches we'll raise
the return value under certain circumstances.

Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Marcel Apfelbaum <marcel@redhat.com>
Cc: Thomas Lamprecht <t.lamprecht@proxmox.com>
Ref: #59
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
lersek added a commit that referenced this issue Mar 23, 2016
@lersek
OvmfPkg: PlatformPei: determine the 64-bit PCI host aperture for X64 DXE
7e5b1b6 
The main observation about the 64-bit PCI host aperture is that it is the
highest part of the useful address space. It impacts the top of the GCD
memory space map, and, consequently, our maximum address width calculation
for the CPU HOB too.

Thus, modify the GetFirstNonAddress() function to consider the following
areas above the high RAM, while calculating the first non-address (i.e.,
the highest inclusive address, plus one):

- the memory hotplug area (optional, the size comes from QEMU),

- the 64-bit PCI host aperture (we set a default size).

While computing the first non-address, capture the base and the size of
the 64-bit PCI host aperture at once in PCDs, since they are natural parts
of the calculation.

(Similarly to how PcdPciMmio32* are not rewritten on the S3 resume path
(see the InitializePlatform() -> MemMapInitialization() condition), nor
are PcdPciMmio64*. Only the core PciHostBridgeDxe driver consumes them,
through our PciHostBridgeLib instance.)

Set 32GB as the default size for the aperture. Issue#59 mentions the
NVIDIA Tesla K80 as an assignable device. According to nvidia.com, these
cards may have 24GB of memory (probably 16GB + 8GB BARs).

As a strictly experimental feature, the user can specify the size of the
aperture (in MB) as well, with the QEMU option

  -fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=65536

The "X-" prefix follows the QEMU tradition (spelled "x-" there), meaning
that the property is experimental, unstable, and might go away any time.
Gerd has proposed heuristics for sizing the aperture automatically (based
on 1GB page support and PCPU address width), but such should be delayed to
a later patch (which may very well back out "X-PciMmio64Mb" then).

For "everyday" guests, the 32GB default for the aperture size shouldn't
impact the PEI memory demand (the size of the page tables that the DXE IPL
PEIM builds). Namely, we've never reported narrower than 36-bit addresses;
the DXE IPL PEIM has always built page tables for 64GB at least.

For the aperture to bump the address width above 36 bits, either the guest
must have quite a bit of memory itself (in which case the additional PEI
memory demand shouldn't matter), or the user must specify a large aperture
manually with "X-PciMmio64Mb" (and then he or she is also responsible for
giving enough RAM to the VM, to satisfy the PEI memory demand).

Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Marcel Apfelbaum <marcel@redhat.com>
Cc: Thomas Lamprecht <t.lamprecht@proxmox.com>
Ref: #59
Ref: http://www.nvidia.com/object/tesla-servers.html
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
lersek added a commit that referenced this issue Mar 23, 2016
@lersek
OvmfPkg: PciHostBridgeLib: install 64-bit PCI host aperture
4f5eff8 
On the normal boot path (which is when PciHostBridgeDxe runs), the PCDs
have been calculated; report the 64-bit PCI host aperture to
PciHostBridgeDxe.

In the Ia32 build, the PCD values (zeros) come directly from the DEC file,
and this patch makes no difference.

Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Marcel Apfelbaum <marcel@redhat.com>
Cc: Thomas Lamprecht <t.lamprecht@proxmox.com>
Ref: #59
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
@lersek
Copy link
Member

lersek commented Mar 23, 2016

Fixed by commits 8f35eb9..4f5eff8.

@lersek lersek closed this as completed Mar 23, 2016
@lersek
Copy link
Member

lersek commented Jul 21, 2016

This (closed) item has been manually migrated to
https://tianocore.acgmultimedia.com/show_bug.cgi?id=80

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ThomasLamprecht @lersek