Simple xocc test fails to compile in hw_emu mode

[j-stephan]

I followed the build instructions in GetStartedWithSYCLCompiler.md and tried to compile the test case mentioned in XilinxFPGACompilation.md. This produces the following error message:

$ /home/ubuntu/software/sycl/compiler/bin/clang++ -std=c++2a -fsycl -fsycl-xocc-device \ 
  single_task_vector_add.cpp -o single_task_vector_add -lOpenCL
In file included from single_task_vector_add.cpp:6:
In file included from /home/ubuntu/software/sycl/compiler/lib/clang/9.0.0/include/CL/sycl.hpp:11:
In file included from /home/ubuntu/software/sycl/compiler/lib/clang/9.0.0/include/CL/sycl/accessor.hpp:14:
In file included from /home/ubuntu/software/sycl/compiler/lib/clang/9.0.0/include/CL/sycl/buffer.hpp:10:
In file included from /home/ubuntu/software/sycl/compiler/lib/clang/9.0.0/include/CL/sycl/detail/buffer_impl.hpp:15:
In file included from /home/ubuntu/software/sycl/compiler/lib/clang/9.0.0/include/CL/sycl/detail/queue_impl.hpp:12:
In file included from /home/ubuntu/software/sycl/compiler/lib/clang/9.0.0/include/CL/sycl/detail/scheduler/scheduler.h:15:
In file included from /home/ubuntu/software/sycl/compiler/lib/clang/9.0.0/include/CL/sycl/device.hpp:12:
In file included from /home/ubuntu/software/sycl/compiler/lib/clang/9.0.0/include/CL/sycl/detail/device_impl.hpp:13:
/home/ubuntu/software/sycl/compiler/lib/clang/9.0.0/include/sycl_wrappers/algorithm:498:47: error: non-constexpr declaration of 'min_element'
      follows constexpr declaration
_LIBCPP_CONSTEXPR_AFTER_CXX11 ForwardIterator min_element(ForwardIterator first,
                                              ^
/usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/stl_algo.h:5610:12: note: previous declaration is here
    inline min_element(_ForwardIterator __first, _ForwardIterator __last)

Similar messages are following for lines 502:47, 507:40, 510:40, 513:52, 516:33, 527:47, 531:47, 536:40, 539:40, 542:52, 545:33, 549:1, 553:1, 556:58, 561:1, 564:42, and 567:42 in sycl_wrappers/algorithm.

The system is a fresh Ubuntu 18.10 installation with SDAccel 2018.3 and a self-compiled XRT (2018.3 branch). I'm using the sycl/unified/master branch which is up-to-date as of writing this issue.

[bader]

@j-stephan, could you try to remove /home/ubuntu/software/sycl/compiler/lib/clang/9.0.0/include/sycl_wrappers/algorithm header and see if it helps, please?

[j-stephan]

@bader, yes, compilation in sw_emu mode worked!

Edit 23 May '19: Removing random in the same directory is also required.

[bader]

@j-stephan, thanks for confirming. We will send a patch to https://github.com/intel/llvm removing these problematic stubs shortly.
I hope @keryell or @agozillon will merge it to this repo afterwards.

[j-stephan]

Great news, thanks!

And a follow-up for the Xilinx folks:

When setting XCL_EMULATION_MODE to hw_emu the compilation will fail:

$ /home/ubuntu/software/sycl/compiler/bin/clang++ -std=c++2a -fsycl -fsycl-xocc-device \
  single_task_vector_add.cpp -o single_task_vector_add -lOpenCL

[...]

****** vpl v2018.3 (64-bit)
  **** SW Build 2405991 on Thu Dec  6 23:36:41 MST 2018
    ** Copyright 1986-2018 Xilinx, Inc. All Rights Reserved.

Attempting to get a license: ap_opencl
Feature available: ap_opencl
INFO: [VPL 60-839] Read in kernel information from file '/home/ubuntu/build/sycl/sycl/test/xocc_tests/simple_tests/_x/link/int/kernel_info.dat'.
WARNING: [VPL 74-49] Failed to read in file /tools/Xilinx/SDx/2018.3/platforms/achromatic during platform validation.
INFO: [VPL 60-423]   Target device: xilinx_u200_xdma_201830_1
INFO: [VPL 60-1032] Extracting DSA to /home/ubuntu/build/sycl/sycl/test/xocc_tests/simple_tests/_x/link/vivado/.local/dsa
INFO: [VPL 60-251]   Hardware accelerator integration...
ERROR: [VPL 60-399] vivado failed, please see log file for detail: '/home/ubuntu/build/sycl/sycl/test/xocc_tests/simple_tests/_x/link/vivado/vivado.log'
ERROR: [VPL 60-806] Failed to finish platform linker
ERROR: [XOCC 60-398] vpl failed
ERROR: [XOCC 60-626] Kernel link failed to complete
ERROR: [XOCC 60-703] Failed to finish linking
/usr/bin/ld: /tmp/single_task_vector_add-6a95df.o: file not recognized: file truncated
clang-9: error: sycl-link-xocc command failed with exit code 1 (use -v to see invocation)
clang-9: error: linker command failed with exit code 1 (use -v to see invocation)

The full compiler log (when calling clang++) and Vivado log can be found here:
compiler.log
vivado.log

[agozillon]

This is one of those unfortunate occasions where I can't seem to recreate the problem on my machine (2018.3 branch XRT, SDx 2018.3 with the build time stamp, same FPGA platform and the same unmodified branch of the compiler), so it doesn't appear to be an issue with the SYCL compiler itself.

So the error in the log is: ERROR: [XSIM 43-3409] Failed to compile generated C file xsim.dir/emu_wrapper_behav/obj/xsim_4.c.

Which is unfortunately a little unspecific (like a lot of the Xilinx tool chain errors unfortunately). From looking around the most likely cause is missing library files in your system for xsim to link against.

So this thread fixed it by installing the build-essential package: https://forums.xilinx.com/t5/Simulation-and-Verification/Vivado-2017-2-Simulation-XSIM-43-3409-Failed-to-compile/td-p/856475

And the following thread shows how to track errors like this down: https://forums.xilinx.com/t5/Simulation-and-Verification/One-solution-to-error-quot-Failed-to-compile-generated-C-file/td-p/865415

The above thread also shows the usage of the settings64.sh file that comes with SDAccel, the current environment setup in the documents is the minimum I needed to run it on my machine with my installed libraries. SDAccel comes packaged with its own libraries though so it's possible the settings64.sh will fix it by linking to the packaged libraries. The downside of using settings64.sh is that it will bloat your environment with a lot of unnecessary variables.

Again sorry about having to play technical support in this case as I can't seem to recreate it on my end for the moment. I am of course happy to help, as if it is a missing environment setup step it should be documented.

@bader Thanks for replying to the issue and helping out. Also thanks for the fix update, we'll merge it asap when we see it merged in!

[agozillon]

Aside from that, once it compiles the newer version of XRT from the master branch of XRT (2.2.0) doesn't play nice with the xcl binary generated from hardware simulation at the moment. Which I'll need to look into and fix in the near future, so I would stick to the 2018.3 branch for now like you are correctly doing!

The XRT from the 2018.3 branch seems to work for the xilinx_u250_xdma_201830_1 but not for the xilinx_u200_xdma_201830_1 at the moment. Some argument buffers are getting inferred to different default DDR banks for each platform (u200 assigns to ddr[1] and the u250 assigns to ddr[0]), which may be some variation of this old error in 2018.2 from the u250 which had a workaround: https://www.xilinx.com/support/answers/71596.html

The version I was previously testing with and that was working reasonably well (in the sense that it was running the executable, I imagine the newer XRT is more accurately simulating hardware and that's the source of the errors) was XRT 2.1.0 from a while ago (Nov 12 15:59:21 2018)!, the commit hash is: e45216d98a2940d5b46534a8f8c8c33dfae6787c So perhaps you'll have more success with the U200 there for the time being...

I think the main take away from this for me is that I need to do some more hardware simulation tests and fixes to account for the newer changes to XRT at the moment..

[j-stephan]

Wow, thanks for the very detailed response(s)!

And the following thread shows how to track errors like this down: https://forums.xilinx.com/t5/Simulation-and-Verification/One-solution-to-error-quot-Failed-to-compile-generated-C-file/td-p/865415

This was the solution! Turns out I had the same issue with Xilinx' bundled libtinfo.so.5 which doesn't play nice with the system's libncurses.so.5. Since the instructions are slightly different for the SYCL compiler I'll document my steps here for anyone else facing this issue:

1. source $XILINX_SDX/.settings64-SDx.sh. Not sure if this is necessary but keeping it here for consistency with the link.
2. Compiling $SYCL_HOME/sycl/test/xocc_tests/simple_tests/single_task_vector_add.cpp creates the directory _x, I assume in the directory from which you have called the compiler. cd to _x/link/vivado/prj/prj.sim/sim_1/behav_gdb/xsim.
3. If you ls here you will see a script called elaborate.sh which you execute in the following way: ./elaborate.sh | grep xelab. This will produce a very elaborate command
4. Copy the mentioned command and append -v 2 to its end, then execute. After some time you should end up with the actual issue and can do some symlinking or whatever.

---

Edit:

The XRT from the 2018.3 branch seems to work for the xilinx_u250_xdma_201830_1 but not for the xilinx_u200_xdma_201830_1 at the moment.

So the correct way to do this would be like in the following link, right? Create a custom OpenCL buffer and pass it on to SYCL?

https://forums.xilinx.com/t5/SDAccel/use-multi-DDR-bank-in-SDAccel-is-not-wrok/td-p/809363

Or is there a nice SYCL way I'm missing?

---

Edit 2: Compilation also worked in hw mode, so I suppose this specific issue is fixed?

[agozillon]

No problem, happy that it helped!

I think I'll leave it open for the moment in case anyone else runs into a similar problem and to remind me to add it to the documentation under perhaps a "Diagnosing Environment Problems" or "Common Problems" section as it seems to be useful information. Our tools can be a little difficult to integrate first time around, the Xilinx trial by fire.

As for the buffer inference problem, I've not had time to look into it just yet (if you find a fix I'd be ecstatic to hear about it), but that's probably one avenue of fixing/working around it, using the OpenCL interoperability components of SYCL. I haven't done a lot of testing with the OpenCL interoperability components of the runtime yet, so I'm unsure how well it will work.

A more long-term and SYCL'y fix (and I still need to find some time to tinker with this to see if it is actually the problem) would be to adjust the sycl/source/scheduler/commands.cpp when it's setting the kernel arguments via clSetKernelArg. That way the user would hopefully never be forced to use the interop components and the compiler would ideally never have to make assumptions.

It may not be as easy as that though, I still need to invest some time into this particular problem and I think my future plans will be to work on cleaning up the driver (I'll talk to @keryell when he gets back from the Khronos F2F next week).

[agozillon]

The problem previously mentioned about inferred DDR Banks should be fixed by this pull request: #27

It's an imperfect fix though, it forces everything into DDR bank 0 for now. Ideally we'd eventually end up with a way for users to easily specify DDR bank and SLR assignment, a future consideration for now though.

[j-stephan]

This seems to be fixed, I can compile in hw_emu mode.