fbc_getCaptureCapability fails with AMD device when both AMD and NVidia cards present

[jamiehayes]

I have both a GTX 1080 and an AMD Vega in my machine. My default adapter is the AMD Vega. When I call fbc_getCaptureCapability, the OS and graphics card checks succeed, and so does the initialization of encoder components, but it fails in EncoderMain::initSessionANDdriverCapabilityCheck because nvidiaDevice is true, and the encoder was created, but the device it was given was not the NVidia device. Hence, when it tries to initialize CNvHWEncoder in NVEncoder::initNVEncodingSession, Initialize() fails with NV_ENC_ERR_UNSUPPORTED_DEVICE.

[jamiehayes]

I changed EncoderMain::checkGraphicsCardCapability with what I think is more proper code and it seems to have fixed this particular issue:

FBCAPTURE_STATUS EncoderMain::checkGraphicsCardCapability() {
    FBCAPTURE_STATUS status = FBCAPTURE_STATUS_OK;

    const int nvidiaVenderID = 4318;  // NVIDIA Vendor ID: 0x10DE
    const int amdVenderID1 = 4098;  // AMD Vendor ID: 0x1002
    const int amdVenderID2 = 4130;  // AMD Vendor ID: 0x1022

    if(device_ == nullptr) {
        DEBUG_ERROR("You must call setGraphicsDeviceD3D11 before you can call checkGraphicsCardCapability");
        return FBCAPTURE_STATUS_SYSTEM_INITIALIZE_FAILED;
    }

    // check the device given for the adapter
    ScopedCOMPtr<IDXGIDevice> dxgidevice;
    ScopedCOMPtr<IDXGIAdapter> adapter;
    DXGI_ADAPTER_DESC adapterDescription;
    HRESULT hr = device_->QueryInterface(IID_PPV_ARGS(&dxgidevice));
    if(FAILED(hr))
        return FBCAPTURE_STATUS_DXGI_CREATING_FAILED;

    dxgidevice->GetAdapter(&adapter);
    if(adapter != nullptr) {
        adapter->GetDesc(&adapterDescription);

        wstring wDeviceName(adapterDescription.Description);
        string sDeviceName(wDeviceName.begin(), wDeviceName.end());
        DEBUG_LOG_VAR("Graphics Card Info: ", sDeviceName);

        if(adapterDescription.VendorId == nvidiaVenderID)
            this->nvidiaDevice = true;
        else if(adapterDescription.VendorId == amdVenderID1 || adapterDescription.VendorId == amdVenderID2)
            this->amdDevice = true;
    }

    if (!this->nvidiaDevice && !this->amdDevice) {
        DEBUG_ERROR("Unsupported graphics card. The SDK supports only nVidia and AMD GPUs");
        return FBCAPTURE_STATUS_UNSUPPORTED_GRAPHICS_CARD;
    }

    return status;
}