Can't compile QAT on Fedora30

[mikeBashStuff]

SPDK: v20.04-rc1-166-gde11a5b2a
OS: Fedora30 (kernel: 5.6.11, gcc: 9.3.1)

An attempt to compile QAT via vm_setup.sh fails with the following:

make -C /lib/modules/5.6.11-100.fc30.x86_64/build M=/QAT/quickassist/qat modules
make[2]: Entering directory '/usr/src/kernels/5.6.11-100.fc30.x86_64'
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_c3xxx/adf_drv.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_c3xxx/adf_c3xxx_hw_data.o
  LD [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_c3xxx/qat_c3xxx.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_c3xxxvf/adf_drv.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_c3xxxvf/adf_c3xxxvf_hw_data.o
  LD [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_c3xxxvf/qat_c3xxxvf.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_c62x/adf_drv.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_c62x/adf_c62x_hw_data.o
  LD [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_c62x/qat_c62x.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_c62xvf/adf_drv.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_c62xvf/adf_c62xvf_hw_data.o
  LD [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_c62xvf/qat_c62xvf.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_cfg.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_isr.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_ctl_drv.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_dev_mgr.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_dev_err.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_init.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_accel_engine.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_aer.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_transport.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_admin.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_hw_arbiter.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_common/qat_uclo.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_common/qat_hal.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_cfg_dev_dbg.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_heartbeat.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_heartbeat_dbg.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_ver_dbg.o
  CC [M]  /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_clock.o
/QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_clock.c: In function ‘timespec_to_us’:
/QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_clock.c:118:17: error: dereferencing pointer to incomplete type ‘const struct timespec’
  118 |  return ((s64)ts->tv_sec * USEC_PER_SEC +
      |                 ^~
/QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_clock.c: In function ‘measure_clock’:
/QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_clock.c:132:18: error: storage size of ‘ts1’ isn’t known
  132 |  struct timespec ts1;
      |                  ^~~
/QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_clock.c:133:18: error: storage size of ‘ts2’ isn’t known
  133 |  struct timespec ts2;
      |                  ^~~
/QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_clock.c:134:18: error: storage size of ‘ts3’ isn’t known
  134 |  struct timespec ts3;
      |                  ^~~
/QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_clock.c:135:18: error: storage size of ‘ts4’ isn’t known
  135 |  struct timespec ts4;
      |                  ^~~
/QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_clock.c:146:3: error: implicit declaration of function ‘getnstimeofday’ [-Werror=implicit-function-declaration]
  146 |   getnstimeofday(&ts1);
      |   ^~~~~~~~~~~~~~
/QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_clock.c:135:18: warning: unused variable ‘ts4’ [-Wunused-variable]
  135 |  struct timespec ts4;
      |                  ^~~
/QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_clock.c:134:18: warning: unused variable ‘ts3’ [-Wunused-variable]
  134 |  struct timespec ts3;
      |                  ^~~
/QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_clock.c:133:18: warning: unused variable ‘ts2’ [-Wunused-variable]
  133 |  struct timespec ts2;
      |                  ^~~
/QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_clock.c:132:18: warning: unused variable ‘ts1’ [-Wunused-variable]
  132 |  struct timespec ts1;
      |                  ^~~
cc1: some warnings being treated as errors
make[5]: *** [scripts/Makefile.build:268: /QAT/quickassist/qat/drivers/crypto/qat/qat_common/adf_clock.o] Error 1
make[4]: *** [scripts/Makefile.build:505: /QAT/quickassist/qat/drivers/crypto/qat/qat_common] Error 2
make[3]: *** [scripts/Makefile.build:505: /QAT/quickassist/qat/drivers/crypto/qat] Error 2
make[2]: *** [Makefile:1683: /QAT/quickassist/qat] Error 2
make[2]: Leaving directory '/usr/src/kernels/5.6.11-100.fc30.x86_64'
make[1]: *** [Makefile:89: modules] Error 2
make[1]: Leaving directory '/QAT/quickassist/qat'
make: *** [Makefile:1560: qat-driver-all] Error 2

Currently, vm_setup.sh uses the following driver package: https://01.org/sites/default/files/downloads//qat1.7.l.4.9.0-00008.tar.gz

Any hints on how to fix this would be appreciated. :)

[peluse]

The QAT driver is not part of SPDK so really we need the issue to be reported to the QAT folks but I believe @karlatec recently worked through something similar so may be able to help. I'll try to find the process for reporting issues

[peluse]

BTW I think the latest driver might work, it's available at https://01.org/intel-quickassist-technology and this site also lists the maintainers contact info.

[mikeBashStuff]

Currently, driver that's listed as the latest one is https://01.org/intel-quickassist-technology/intel-quick-assist-technology/downloads/intel%C2%AE-quickassist-technology-driver-linux-hw-version-1.7-l.4.9.0-00008 and that's what vm_setup.sh is actually using. :)

[peluse]

Yeah that's probably from when @karlatec updated it recently. We're not using Fedora30 in CI yet are we?

[mikeBashStuff]

I just logged onto one of the random Fedoras in the pool and it actually was the Fedora30 (though idle, so maybe it's not picking up any builds yet. :)).

[peluse]

or it might not be one used to test QAT....

[karlatec]

@mikeBashStuff which QAT system is that?
Issue title is probably misleading. You can compile & install QAT driver on F30. I know that, because I installed this particular driver version a couple of times, no problems. The difference is that I compiled it on 5.5.10 kernel with 9.2.1 gcc, so that's probably the cause.
You should probably report that on 01.org and wait for new driver version.

[mikeBashStuff]

@karlatec It was done on my local VM.

Since we support kernel upgrades (since that's what vm_setup.sh can do) we should also handle it in some way rather then just waiting for the new driver (taking for granted that newer kernel|compiler is the culprit here).

[karlatec]

we should also handle it in some way rather then just waiting for the new driver

Explain?

[mikeBashStuff]

@karlatec If the kernel version makes any difference, or the gcc, we shouldn't try to compile qat stuff but simply return message stating that it's not supported in such a setup, i.e., be graceful about it.

[karlatec]

I don't think it's our responsibility to check which kernel and gcc versions are supported/correct for various QAT driver versions.

[mikeBashStuff]

@karlatec Ok, closing this issue then. :) If you have any knowledge on how stuff like that should be reported to 01.org then please share (I can't see any QAT references under their mailing lists|github repos|bug tracking).

[peluse]

@mikeBashStuff I think I mentioned this earlier but if you look on the QAT page the maintainers are listed there. I don't believe they run a community like SPDK does but there are some contact names there fore sure.

[mikeBashStuff]

@peluse I looked into that list, but from what I saw, only names|usernames were listed there, no contact info of any sort. :) And maybe it's silly, but I usually try to avoid contacting maintainers from outside projects directly - I once tried that by contacting one of the Linux kernel maintainers, via his email address, and let's just say his response to me doing that wasn't the very essence of politeness. :P

[peluse]

@mikeBashStuff oh geeze. Let me ping someone internally and find out what the best channel is to get support

[peluse]

@mikeBashStuff from a reliable source: just email the dpdk folks at users@dpdk.org

[mikeBashStuff]

@peluse Thanks, will try that!

[mikeBashStuff]

Will put my finding here as well, just in case (this is a message I sent out in reply to users@dpdk.org).

Appreciate your response!  The thing is that this issue suddenly popped up after the kernel upgrade - I just quickly kexeced into the older version (5.5) and QAT driver was compiled against its source without any hiccups. 
By briefly diffing two kernel sources, it seems that it relates to changes introduced in timekeeping32.h and time.h. In the former, getnstimeofday(), that's used by QAT driver, is not declared anymore. As for the latter, #ifndef __KERNEL__ now hides the time* types. Commits that introduced this:

c766d1472c70d25ad475cf56042af1652e792b23 - time.h
412c53a680a97cb1ae2c0ab60230e193bee86387 - timekeeping32.h

So I think some changes needs to be done in the actual driver code. But I will try to reach out to folks at 01.org.

And to note that if the above is correct then this issue is not limited to Fedora's kernels, but all the ones that simply are more recent.

[Ornias1993]

@peluse If you like you can just openly admit QAT is vaporware, just so we can all start dropping it.
It has been months and QAT is STILL not actually supporting kernel 5.6 and up.

[peluse]

@Ornias1993 I don't see your name in the history on this, are you also having issues?

[Ornias1993]

@peluse The same one here... QAT not being 5.6 compatible.
Does my name have to be in an issue before an obvious kernel incompatibility gets fixed?! >.<

"Just email folks that do jackshit" isn't really good support of your hardware.

[peluse]

@Ornias1993 OK, thanks. And no, of course your name doesn't have to be on it it to get it fixed :) It does help us (the SPDK team) communicate with the QAT team that it's more than the SPDK CI environment that's being impacted so appreciate you speaking up! I've escalated this internally, stay tuned...

[Ornias1993]

@peluse Even a stock build (with --enable-kapi) on 5.6 doesn't pass.
Which is pretty reasonable, because it seems the functions it crashes on have been changed in 5.6.

If they tested it on 5.6 (which they should've been from feb 2020 (rc1) onwards at the very least) they would've known.
OS tested: Deb with kernel 5.6

[Ornias1993]

Thanks to the patch file you guys from spdk have provided I managed to continue a but further.
Thank you!