-
Notifications
You must be signed in to change notification settings - Fork 145
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create example compiling and running in raspberry pi with Mesa Vulkan drivers #131
Comments
There is a blog post from June 2020 on how to get up and running with V3DV driver which I think would be a good starting point: v3dv: quick guide to build and run some demos – infapi00. Although it looks like some of the dependencies might be superfluous for running Vulkan Kompute. Perhaps a more reusable example would come from being able to deploy Vulkan Kompute to a Raspberry Pi 4 using a service such as balena. Balena is used to deploy docker containers to IoT devices. The underlying operating system is balenaOS which is open source. A working Dockerfile which can be deployed at ease to an IoT device would make this library very attractive for those doing Edge processing. On the topic of using Docker, I noticed the Vulkan Kompute Dockerfile makes use of the nvidia/vulkan base image. This image in turn is a combination of glvnd/devel/Dockerfile and vulkan/Dockerfile. Perhaps the latter Dockerfile implementations, coupled with one of the Balena base images, would be good a good starting point for getting this deployed as a container to a Raspberry Pi. I realise this comment is a single perspective given the tools I am familiar with from work (the keen reader might have noticed Balena mentioned a couple of times). I do not work for Balena, but the company I work for makes extensive use of their services. Any feedback on the above thoughts would be most welcomed. |
@hpgmiskin thank you for the comments! Sounds really interesting - for the example it may be best to have the simplest initial implementation, which could allow for one of the examples to run in a Raspberry Pi. Having a Docker image may be worth it as well, but ideally it could be developed such that the script to set up the dependencies can be separate (or on a Makefile). Ultimately you can see the dependencies required to run Kompute from scratch (whcih really are just The base docker image that may be the best one to take as example would be the base runner - there are tons of dependencies that are installed and set up that you wouldn't need to (pretty much you only need the things outlined above, after setting up the drivers) https://github.com/EthicalML/vulkan-kompute/blob/master/docker-builders/KomputeBuilder.Dockerfile I would suggest having an example like https://github.com/EthicalML/vulkan-kompute/tree/master/examples/logistic_regression where the location is relative to the repo, and everything can be built by just running Happy to share further details on each of these points. Most of the points of the blog post are quite general, except the point where it uses the custom ICD (ie There is quite a lot of nuanced terminology on this, but perhaps the easiest is if you are up for it, you can create a fork, add a folder, and give it an initial shot - if you have any errors these probably would be either: 1) straightforward errors we've come across before, or 2) obscure errors that are related to v3dv for which we can reach out to the mesa team. |
In some preliminary testing Vulkan Kompute is working on a Raspberry Pi 4. Starting from 2021-01-11-raspios-buster-armhf and following v3dv: quick guide to build and run some demos – infapi00 the vulkan-kompute/test_array_multiplication.py test passes when run on a Pi 4. I am going to pair back the dependencies to the bare minimum in order to build mesa and create an example in the examples folder with a README.md documenting the steps required. |
@hpgmiskin awesome! That would be great, let me know if you have any questions or if you need a hand - I just did a pretty significant refactor so it may be best to actually build directly from master instead of the 0.6.0 branch. If you get stuck or see any issues, just post the debug logs and I'll be able to point you in the right direction. |
I got some time to experiment with this today. I will open a draft PR in order to share the (fairly rudimentary) README.md in the example folder. Currently only 8/12 tests are passing on master branch. I think some of the failing tests might be due to running on a 32 bit operating system so float64 is not available.
As for next steps:
Any feedback on PR or next steps much appreciated. |
@hpgmiskin thank you for the heads-up! Interestingly enough I have added a feature this weekend which will allow us to explicitly circumvent this. The issue that you currently have is not specifically the architecture, it is just that many systems actually don't support float64 (which interestingly enough is Python's default - they use It seems a few tests actually pass, so it seems the issue may just be with some of the more particular features which often may not be supported in Mesa drivers. If you look at the tests that has that error is basicallyy the Overall this seems quite positive tho! It would be good to understand what are the more specific features that are not working, could you re-run the tests but share all the debug logs? You can do this by running the tests with the following:
Thank you for having a look at this Henry and for the PR, looks great - it looks great, I will actually reference it in the main documentation, similar to how we just add the README of the android example https://kompute.cc/overview/mobile-android.html |
Here is the full log output from running pytest. Interesting the number of failed tests varies from 3 to 8 when I run the test suite multiple times. Could there be some sort of race condition which is making the test pass rate different? |
Ok very interseting, thank you for sharing the logs - it indeed looks exactly like a race condition. It seems that when we run multiple |
@axsaucedo running the tests on #182 seems to mostly solve the issues. There was one occasion where test array multiplication failed. However, I was not able to reproduce this failure, even when using pytest-flakefinder. I will try a couple more times to see if something comes up. As expected the double test was still failing. I will add a similar pytest skip, as is used with swiftshader, in order to have a fully passing test suite. When #182 is merged I can merge upstream master into this branch in order to update and test the latest Python test suite. |
THank you @hpgmiskin, sounds quite positive! Ok great, I am adding an |
@hpgmiskin I have added another change to #182 and trying to investigate if there are any memory barriers that are set incorrectly or missing - I realised that the Staging memory was not using a HOST_COHERENT bit, which in the case the memory wasn't coherent we would have to flush the memory (which we don't currently do as we currently restrict to only coherent memory heap). I've gotten some further insights that the Pi heap memory is always coherent so this may not make difference but so far it does seem like an important piece. If you get a chance would be great if you can test and see if you can replicate the failure of the |
@axsaucedo I have installed and run the tests against the latest commit on #182, the only test to fail is the test type double. Even with running many replicate test there is no issue with the multiplication test. |
Awesome - thank you for taking the time to test @hpgmiskin, this sounds quite positive so I'll merge now. I'll do some further investigation to ensure there are no discrepancies with the memory barriers but it seems this PR does address quite a lot of issues that would've appeared down the line. THanks! |
@axsaucedo I will merge in master later this weekend and update the tests to ignore double on broadcom GPU. There is just one element it would be good to get your thoughts on. When testing the other day I realised an issue with ssh access to the device. Previously I had been using remote login while also logged in through a display, therefore, remote access to the renderer worked. When attempting to run the examples through remote login while the login page was displayed on the conventional screen resulted in an error.
This corresponds with experiences discussed on [Mesa-users] Vulkan VK_ICD_FILENAMES and /dev/dri/renderD128 permissions where the render access is not allowed until a non remote user has logged in.
Is this something that you have had any experience of? I could attempt to brute force using |
I don't think modifying permissions is always bad practice, it just depends whether it's being done in a way that is consistent with the expected access to the device and files - you should also be able to use |
Thanks for the hand holding and sorry for all the questions. Running
Then running |
No worries - great to hear @hpgmiskin ! |
Although looking at this I do think the best practice would be having the driver still owned by root, and using |
I have made some updates to the PR which should hopefully close out this issue 🤞 |
Awesome @hpgmiskin ! It does - awesome contribution! Just merged! Let me know your thoughts later on if you get any ideas for a potential follow-up, the edge detection use-case sounded really interesting |
It seems there are some relatively recent advancements in the Vulkan Drivers support for RaspberryPis (https://www.raspberrypi.org/blog/vulkan-update-were-conformant/). This issue encompasses exploring putting together an end to end example similar to the android example that shows how to run Kompute on a Raspberry Pi using the Mesa driver which enables for Vulkan 1.0 compliant processing in the Raspberry Pi (https://gitlab.freedesktop.org/mesa/mesa)
The text was updated successfully, but these errors were encountered: