How to integrate a new programmer

[aalku]

How can I add a new programmer?

I see commands on "programmers.json" but I don't know where to place the executable (a python script in my case) so apio finds them. I can put it on the system path on my computer but I don't know how it's supossed to be distributed or installed with apio.

I can see Icestudio creates a virtual environment and some programmers are there but it seems wierd to me to have the "programmers.json" in apio and the actual executables in the project that is going to use apio. If I just make pull request of the programmers.json noone in this project can test it if the programmer is not here or referenced.

I'm sure there's something im not getting right. Thank you!

[Jesus89]

Hi. Here you have a little document regarding new boards support: https://apiodoc.readthedocs.io/en/stable/source/contribute/support_new_board.html.

The integration of the programmer depends on the type of programmer:

• Python package: this requires publishing the package into Pypi, with a version number, and then include it in the Apio's pip_packages list. An example of this one is tinyprog: https://pypi.org/project/tinyprog/.
• C/C++ package: this requires publishing a toolchain with the code built for different architectures in static. For example the iceprog programmer: https://github.com/FPGAwars/toolchain-ice40/releases. There's more information about the build scripts in this repo: https://github.com/FPGAwars/toolchain-icestorm.

Sometimes the same programmer (for example iceprog) is used in different boards. In your case (icesugar) seems to work with iceprog programmer too, and the FPGA is already included in the fpgas.json file. So it only requires adding the board in the boards.json file.

[aalku]

Thank you for your complete answer!

I don't think iCESugar supports iceprog. It has a tool called icesprog but no source or windows binary has been released so far. The other upload method it to copy the bin to a virtual flash drive. It's quite fast but I don't know how to write with an specified offset (for risc-v firmwares). I'm thinking about using an external SPI programmer. I'm already using an external USB-UART...

But for just uploading one single bin to default offset (0?) the copy method works great.

Again, thank you for your time.

[aalku]

The author just released icesprog source. I'll try to make a windows build with mingw and then create a toolchain.

[Jesus89]

You're right. icesprog != iceprog

Glad to see a new toolchain coming 😄

[aalku]

The author published (by request) icesprog source and I got to build it for windows. I guess a new toolchain is coming 😊

[aalku]

I have a lot of problems getting this to work.
I can't get icestudio to build installers or be installed or started from source directory but that's another story.
So I installed icestudio with the installer, replaced apio files on it's toolchain zips.
Then I install the toolchain and it install my new icesprog package BUT.

The installer binary lands at: C:\Users\user\.icestudio\apio\packages\icesprog\bin\icesprog.exe

And apio doesn't look for it there:

C:\Users\user\Desktop\fpgas>"C:\Users\user\.icestudio\venv\Scripts\apio.exe" upload --board iCESugar_1_5 --verbose-pnr -p "C:\Users\user\Desktop\fpgas\apio-icesugar-project-test"
[Sun Oct 18 22:13:59 2020] Processing iCESugar_1_5
------------------------------------------------------------------------------------------------------------------------
icesprog hardware.bin
"icesprog" no se reconoce como un comando interno o externo,
programa o archivo por lotes ejecutable.
scons: *** [upload] Error 1
============================================= [ ERROR ] Took 1.50 seconds =============================================

How can I reference that path from programmers.json or fix it other way?

Programmer package is here https://github.com/aalku/apio-toolchain-iCESugar.
Apio fork here https://github.com/aalku/apio/tree/develop-iCESugar.
And by now its outside of the scope but icesudio fork is here https://github.com/aalku/icestudio.

Edit: I'm starting with windows_x64. I wish to cover every or most plaforms.

Thank you in advance.

[aalku]

It seems the programmer must be added to path in util.py.
I'm sure it's needed for spacial cases but I think package bin directory might be added to path from json definition files. I don't see the need of changing code (util.py) to add a package that is just basic.

I got it working :)

I would make a pool request now and then add the other platforms later but maybe you want to process a single poll request with everything instead.

[Obijuan]

Great! it is up to you. You can do the pull requests you like. It is not a problem

[Obijuan]

Let's continue the integration on this issue.

First step: Create the toolchain. Please, provide here the link to the repo of the upstream you want to include, so that we all can check it

[aalku]

Thank you. This is the repository of the whole board. https://github.com/wuxx/icesugar

The programmer software is in src. As you can see there are binaries for different architectures of Linux but there aren't for windows. I asked for the sources and the author published them. They are on tools/src.
I got to build the windows binary myself and I host it on a repository of mine.

[Obijuan]

OK. As the upstream does not provide a Release with the tools yet, we have to include the building script in the icesugar apio package. Remember that apio is a package manager, like pip,npm or apt-get. It installs the package that the user wants (The user tells explicitly apio the package to install. It does not install anything without the user consent)

Next step: Create the toolchain-icesprog repo. It will contains the binaries for all the supported platforms as well as the scripts for building from the upstreams sourcer or downloading oficial releases if the upstream has already compiled them

I will create the repo. Just give me some time. In the meanwhile you can have a look at other apio packages, like https://github.com/FPGAwars/tools-system (We should create something similar. Everything should be statically compiled, so that no external libraries are requiered)

[aalku]

Hi. Let me tell you my progress. :)

I'm copying things from the scripts of that FPGAwars/tools-system repo and by now I get to compile libhidapi and it's example statically for windows_x86, windows_x64 and linux_x64 with the same script set. It was hard but I expected it to be harder. I have no experience creating build scripts myself.

I guess it will be easy to compile icesprog statically with them.
Other platforms currenty fail but I think windows_x64 and linux_x64 are the most important ones so this is an important advance.

The script set downloads icesugar, libusb and libhidapi and compiles the libraries and their tests and then checks that they are statically linked.
The next thing will be compile icesprog with them.

I think once I can compile icesprog and build the package tar.gz files with just calling ./build.sh <arch> we can add it to the upcomming https://github.com/FPGAwars/toolchain-icesprog and we are ready to work integrating it to apio if you think that's enough.

[Obijuan]

Great!

I've already created the repo for the apio package:

https://github.com/FPGAwars/toolchain-icesprog

The build script should be similar to the one created in

https://github.com/FPGAwars/toolchain-fujprog

but with the adaptations needed

[aalku]

Nice!

I succesfully build the icesprog binary from upstream sources :) for linux_i686 too.

But I don't know if this is a problem. I don't use icestudio on Linux. It seems the programmer needs root access.

What do you think? Is that a problem?

I'll build the gzip packages, fork FPGAwars/toolchain-icesprog repo, upload everything to the fork (no binaries at all), maybe test build from clean ubuntu (18.04) with no previous dependencies installed, and finally pull-request. Please feel free to reject it if I missed any step or you have something else in mind.

But @Obijuan, toolchain-fujprog is only packaging preexisting binaries, I am compiling them too. The final result should be the same, a package with the same format, but with different way to get the binaries first.

Note: One good thing about icesprog is that it doesn't need drivers to be installed on windows. I think it's that hidapi. I wonder if iceprog could use it too. I have no idea.

[aalku]

I've got this tag and release with binaries compiled by myself with the included scripts.

https://github.com/aalku/toolchain-icesprog/releases/tag/v1.0.0d

I'll try them with apio and maybe with icestudio (if I still can build it).

Then I will make a branch referencing FPGAwars/toolchain-icesprog in package.json instead my own repository and make pull-request. (No binaries, I understand you will build them???)

[Obijuan]

Great!
I was able to generate the binaries for windows_amd64 and linux_x86_64 in ubuntu 20.04. I've changed the scripts so that they are similar to the ones used in tools-systems.

Could you please pull these changes and test them? I've seen that you introduced some workarounds. Try first to compile with the new scripts and when everything is ok, add the workarounds (in compile_icesprog.sh)

In this first stage we are focusing on the binaries generation

I think that it is great that no drivers are required in windows! I have to see if it is possible to do the same with iceprog. Thanks!

[aalku]

Great! Thank you! Of course!

[aalku]

On Ubuntu 20.04 just installed windows_amd64 is not compilng with these scritps.

+ autoreconf --install --verbose --force
./bootstrap: 2: autoreconf: not found
/home/ubuntu/toolchain-icesprog/scripts/compile_hidapi.sh: line 33: ./configure: No such file or directory
make: *** No targets specified and no makefile found.  Stop.
make: *** No rule to make target 'install'.  Stop.

----------> COMPILAR EJEMPLO!!!!
 x86_64-w64-mingw32-gcc -o hidtest.exe hidtest.cpp -static -L/home/ubuntu/toolchain-icesprog/_builds/build_windows_amd64/hidapi-hidapi-0.9.0/release/lib -I/home/ubuntu/toolchain-icesprog/_builds/build_windows_amd64/hidapi-hidapi-0.9.0/release/include/hidapi -lhidapi-libusb -L/home/ubuntu/toolchain-icesprog/_builds/build_windows_amd64/libusb-1.0.22/release/lib -lusb-1.0 -lpthread -lsetupapi
hidtest.cpp:21:10: fatal error: hidapi.h: No such file or directory
   21 | #include "hidapi.h"
      |          ^~~~~~~~~~
compilation terminated.

I took a note here:
FPGAwars/toolchain-icesprog@d7f21c6#r43557958

After running sudo apt install autoconf the first error is gone but it's still not compiling hidapi I think.

This is the output I don't fully undestand:

make  all-recursive
make[1]: Entering directory '/home/ubuntu/toolchain-icesprog/_builds/build_windows_amd64/hidapi-hidapi-0.9.0'
Making all in windows
make[2]: Entering directory '/home/ubuntu/toolchain-icesprog/_builds/build_windows_amd64/hidapi-hidapi-0.9.0/windows'
make[2]: *** No rule to make target 'all'.  Stop.
make[2]: Leaving directory '/home/ubuntu/toolchain-icesprog/_builds/build_windows_amd64/hidapi-hidapi-0.9.0/windows'
make[1]: *** [Makefile:522: all-recursive] Error 1
make[1]: Leaving directory '/home/ubuntu/toolchain-icesprog/_builds/build_windows_amd64/hidapi-hidapi-0.9.0'
make: *** [Makefile:415: all] Error 2
Making install in windows
make[1]: Entering directory '/home/ubuntu/toolchain-icesprog/_builds/build_windows_amd64/hidapi-hidapi-0.9.0/windows'
make[1]: *** No rule to make target 'install'.  Stop.
make[1]: Leaving directory '/home/ubuntu/toolchain-icesprog/_builds/build_windows_amd64/hidapi-hidapi-0.9.0/windows'
make: *** [Makefile:522: install-recursive] Error 1

and I think there are not compiled objects in _builds/build_windows_amd64/hidapi-hidapi-0.9.0/**

[aalku]

./build.sh linux_x86_64 works and after that ./build.sh windows_amd64 works too.

I did rm -rf _* and ./build.sh windows_amd64 still works so the problem must be another dependency that was installed for linux and is needed for windows too I think. I'll test the windows binary now so I can tell you if the workarround is still needed or not but there's something missing in the windows dependencies and I don't know what.

[aalku]

The windows_amd64 binary is not working :(

The problem's in the upstream code with Windows.

Here is where it fails:
https://github.com/wuxx/icesugar/blob/6424e0d8f8a48fe0bd77059bbc0fa9bf72767708/tools/src/icesprog.c#L659

read is supossed to read the whole file and that is st.st_size bytes but it get's less bytes so the program chashes.

This problem is described here for the case of cygwin. Not exactly the same on mingw but it's the same:
https://www.cygwin.com/cygwin-ug-net/using-textbinary.html

Windows C library opens the file in text mode so converts what it thinks that are line break codes.
The cygwin solution is on a libbinmode.a to select defalty binary mode that mingw does not have but it does have /usr/x86_64-w64-mingw32/lib/binmode.o but it doesn't work.

Here someone explains that on mingw has a bug with that.
https://sourceforge.net/p/mingw-w64/bugs/693/

I copied this Demyan Kimitsa's code. I think I should've referenced that post in a comment.

That fix is needed so mingw uses binary mode by default.

[aalku]

Pull request made to the toolchain with all these things. If you accept it next step would be adapt apio to it but apio needs binary packages for the toolchain as far as I know so either you build them and publish them or I refrerence mines...

[Obijuan]

We will continue this discussion on This issue and on the FPGAwars group (I will open a new thread). I need you to perform some tests, as I do not have an icesugar board