on-the-fly decompression and spi-over-jtag #47
Comments
|
Hi
|
|
I propose to
support zlib on-the-fly decompression inside of openFPGALoader
and for flashing first search for spioverjtag<idcode>.bit.gz
on-the-fly decompress .gz using zlib support and program it and
then flash like normal.
It will significantly reduce spioverjtag bitstream size in the binary
installation of openFPGALoader
and I also propose to support reading bitstream from standard input,
so that user can do for example:
zcat bitstream.bit.gz | openFPGALoader
…On 9/4/20, Gwenhael Goavec-Merou ***@***.***> wrote:
Hi
It's not clear for me:
- You propose to store spiOverJtag bridge in gzip format, uncompress this
file before loading in RAM ? or
- sending the gzip by spiOverJtag, uncompressing this one in the FPGA before
writing the FLASH?
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#47 (comment)
|
|
Xilinx FPGAs already supports a compressed bitstream format that can be loaded with openFPGAloader as usual. |
|
HI
I know xilinx lattice and others support compressed bitstream,
but additional compression efficiency (file even shorter) can be
achieved by compressing that with gzip -9 and that was the
reason I was asking for this feature.
Instead of this, alternative would be to support reading bitstream
from unix pipe (stdin) so I can use for example
zcat bitstream.bit.gz | openFPGALoader
…On 11/26/20, GpGb ***@***.***> wrote:
Xilinx FPGAs already supports a compressed bitstream format that can be
loaded with openFPGAloader as usual.
To enable this feature it should be sufficient to just set ```set_property
BITSTREAM.GENERAL.COMPRESS TRUE [current_design]``` constraint. Other than
saving space this can make the flashing operation slightly faster.
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#47 (comment)
|
|
I'm agree with both proposition.
Clearly gzip applied on non compressed file is smaller, but with both the flashing time is faster + file small For unix pipe, my problem is a bit different: I use file extension to determine file type (it's diffcult or impossible to determine by content itself), with this approach it's required to find a way to select correct bitstream parser without extension information |
|
Me as unix-based user am inclined to a function independent of filenames and
file content, just have commandline options to control what is done.
There can be user-friendly way which just works but should be allow to
completely
be turned off.
I think imagemagick "convert" is one of counter-examples what should
not happen, when a knowledge of filename extension, content and commandline
options is all required for user to know in order to get wanted result
…On 11/26/20, Gwenhael Goavec-Merou ***@***.***> wrote:
I'm agree with both proposition.
A small test with xilinx compression and/or gzip shows:
| size | compression |
|--------:|--------------------|
|2192129 | without xilinx compression |
| 236791 | with xilinx compression |
| 3390 | without xilinx compression + gzip -9 |
| 15910 | with xilinx compression + gzip -9 |
Clearly gzip applied on non compressed file is smaller, but with both the
flashing time is faster + file small
The question is to determine which library to use to be compatible with
linux, macOS and windows.
For unix pipe, my problem is a bit different: I use file extension to
determine file type (it's diffcult or impossible to determine by content
itself), with this approach it's required to find a way to select correct
bitstream parser without extension information
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#47 (comment)
|
|
It's true, sometime letting deduce behaviour automatically is not a good thing. I will let nonetheless automatic detection for backward compatibility. If someone can tell me an gzip library compatible with linux, windows and macOS ? |
|
right, passing filename extension should make it for pipe
from google search "zlib" seems available on all 3 platforms
but I have tried it only on linux
…On 11/28/20, Gwenhael Goavec-Merou ***@***.***> wrote:
It's true, sometime letting deduce behaviour automatically is not a good
thing.
I think adding option to provides file format it's a good idea and the only
solution to allows using unix pipe.
I will let nonetheless automatic detection for backward compatibility.
If someone can tell me an gzip library compatible with linux, windows and
macOS ?
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#47 (comment)
|
|
My problem is to be sure for all 3 platforms :-) |
|
Hmmm I see the situation, however if in doubt of compatibility
I suggest just to leave pipe function and rely on system to decompress
to stdin. I'm not expert but after 30yr of development wynd0uz
could have some solution to mimic pipes.
…On 11/28/20, Gwenhael Goavec-Merou ***@***.***> wrote:
My problem is to be sure for all 3 platforms :-)
Currently the documentation to install under windows and macOS is missing,
if at least I don't break everything that would be good :-)
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#47 (comment)
|
|
Sorry for delay. |
|
If you can also compile bitstream for artix100, that's on my boards
so this is directly testable here
…On 12/9/20, Gwenhael Goavec-Merou ***@***.***> wrote:
Sorry for delay.
In a first time I have used solution suggested by @GbGp to reduce bitstream
size (and reduce load time): the artix50t bitstream is compressed, others
are left unchanged to avoid noise by updating all of them
I'm currently doing some test to refactor some classes to be able to use
stdin.
For file type as argument, again it's in pipe: this way is sometime required
(for xilinx when using .bin file it's not possible to deduce if it's simple
raw data or a bitstream).
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#47 (comment)
|
|
xc7a100 bitstream is already available. No ? |
|
Yes I forgot, maybe it needs small gzip -9 only,
from 3.8MB to 4KB
…On 12/9/20, Gwenhael Goavec-Merou ***@***.***> wrote:
xc7a100 bitstream is already available. No ?
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#47 (comment)
|
|
Sorry for the long delay. Thanks |
|
HI
Hey this is a great news! I tested new pipe option at ulx3s
and it works good and you have speeded up ft231x somehow
speed is now few % faster than fujprog (was about 50% slower before)
next week I could try some other boards too
…On 2/24/21, Gwenhael Goavec-Merou ***@***.***> wrote:
Sorry for the long delay.
`openFPGALoader` has now `--file-type` to force format (instead of using
file extension) and pipe support.
It's now possible to use cat or gzip | openFPGALoader [options] --file-type
xxxx
Thanks
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#47 (comment)
|
|
Thanks for you feedback. I will reread this issue and the other one to see if I have lost some improvements. And need to find a universal library to uncompress file on the fly |
|
I also don't know if anything faster is possible.
fujprog can't get much speed from ft231x/ft232r than the approx
same speed what you get now and this is effectively 500kbps
(not really 3Mbps as openfpgaloader reports). Maybe ftdi chip is
initialized for 3Mbps serial speed but it bitbangs jtag much slower
jtag bitbang method is fragile with compatibility and can
trigger odd ftdi behaviour for some pc motherboards usb interfaces
and/or different platforms linux/mac/win so we can be happy
if it just works reliably
…On 2/25/21, Gwenhael Goavec-Merou ***@***.***> wrote:
Thanks for you feedback.
Yes I've take time to improve bitbanging speedup (I'm not sure if it's
possible to increase more).
I will reread this issue and the other one to see if I have lost some
improvements. And need to find a universal library to uncompress file on the
fly
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#47 (comment)
|
|
The speed reported by openFPGALoader is internal speed (I have tested with a logic analyzer) to consume FIFO's content, this why it's used. But it's true, for the full bitstream transfer the communication and the small internal FIFO size, will reduce the real speed. |
|
This is from #132 by @trabucayre:
With the need to have access to the tools, I think it would be more convenient to a user especially on windows/mac if it has been compiled into the code using a library, rather than openFPGALoader having to find the executable at runtime. Another alternative might be a small (~20 line) custom C compression/decompression using the basic Lempel-Ziv strategy. This could then be included directly in the openFPGALoader sources. Here is one example: https://github.com/andyherbert/lz1 (There is no license statement however, but I guess the author could be contacted; @andyherbert.) I also tried to do an implementation just from the description given. Such an approach seems to be able to reach a few kB for the .bit files, and 50kB - 200kB for .rbf. This is not worse than a factor of 2 or 3 compared to .gz for the .rbf files. |
|
To avoid binary heavyness, if they are on main source repository
to gzip them first with 4K block size, low-ram decompression friendly
I have a python code here
https://github.com/emard/esp32ecp5/blob/master/gzip4k.py
Other option is to create another repository
github.com/OpenFPGALoader-bin
that contains spioverjtag binaries
…On 11/15/21, Håkan Johansson ***@***.***> wrote:
This is from #132 by @trabucayre:
> My worry about spiOverJtag directory size is primarly to the
> download/clone step. Xilinx bitstream, with compress option enabled, have
> an acceptable size, but intel/altera ones must have compress option
> disabled, and consequently have a size more important. Most globally, due
> the potential number of devices this directory may become awfully heavy.
>
> Concerning decompression at install step vs on-the-fly decompression:
>
> - the first is the most simple. No openFPGALoader's internal modifications
> to do, but requires to have access to the correct tools (gunzip/xz) for
> all systems (possible for windows and mac ?)
>
> - the second only requires to have correct library (again windows/mac ?),
> no modification at install step but maybe increase programming time due to
> decompress step (it's maybe a wrong question, maybe impact is really small
> compared to spi flash transactions).
With the need to have access to the tools, I think it would be more
convenient to a user especially on windows/mac if it has been compiled into
the code using a library, rather than openFPGALoader having to find the
executable at runtime.
Another alternative might be a small (~20 line) custom C
compression/decompression using the basic Lempel-Ziv strategy. This could
then be included directly in the openFPGALoader sources. Here is one
example:
https://github.com/andyherbert/lz1
(There is no license statement however, but I guess the author could be
contacted; @andyherbert.) I also tried to do an implementation just from
the description given. Such an approach seems to be able to reach a few kB
for the .bit files, and 50kB - 200kB for .rbf. This is not worse than a
factor of 2 or 3 compared to .gz for the .rbf files.
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#47 (comment)
|
|
It's true, The best solution, in all case is to compress bitstream in the repository to reduce size and download/clone time. When I've said "an external tool to uncompress file" it's not at runtime but at install time: all bitstream are compressed in repository and instead to simply copy files, bitstream are uncompressed to /usr/share/openFPGALoader. To resume the two options (but maybe others solutions are possible. I'm, of course, open, to all advice):
First one has my favor (maybe I'm wrong) because it has no impact on runtime/speed. Of course, it's possible to select one option and finally to move to another if, finally, it's not the best. |
|
Maybe the easiest option is to decompress spioverjtag
at install time and let openFPGALoader use them uncompressed.
I find on-the-fly decompression inside openFPGALoader good to have
linux is easy, but as we discussed earlier, cross-platform compatibility
could bring problems if we have to rely on external library.
Maybe we can just include simple C++ file that only decompresses
maybe something like this or similar (I haven't tried)
https://bestofcpp.com/repo/richgel999-miniz-cpp-compression
…On 11/15/21, Gwenhael Goavec-Merou ***@***.***> wrote:
It's true, The best solution, in all case is to compress bitstream in the
repository to reduce size and download/clone time.
Using a second repo dedicated to binary files may be a solution but without
compression clone time will be the same as current solution, and it's maybe
less user friendly since it's required to fetch submodule (but I keep this
option in my mind).
When I've said "an external tool to uncompress file" it's not at runtime but
at install time: all bitstream are compressed in repository and instead to
simply copy files, bitstream are uncompressed to /usr/share/openFPGALoader.
To resume the two options (but maybe others solutions are possible. I'm, of
course, open, to all advice):
- bitstream are extracted are install time using external tool,
openFPGALoader is not modified to add uncompress support. This option have
no impact to time to write bitstream
- bitstream are uncompressed at runtime using an external, or internal,
library. This option may increase a bit global runtime (really? compared to
the spi Flash protocol?) to write bitstream, but everything is known since
the library become a required dependency.
First one has my favor (maybe I'm wrong) because it has no impact on
runtime/speed.
Of course, it's possible to select one option and finally to move to another
if, finally, it's not the best.
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#47 (comment)
|
|
Runtime penalty for decompression would be very small. E.g. zcat of 233 kB .gz file containing all the .bin and .rbf files (to 31 MB) takes 0.14 s on a fairly old AMD Athlon. That would be ~230 MB/s. Programming my EP4CE15 with 511 kB takes 1.2 s, i.e. 0.42 MB/s. |
|
Programming time depends, partially, to the probe: arty @30mhz takes 0,735s. I think having one .gz by bitstream is more easy and well VCS compliant (this avoid to update all the time one file, but just add new one). |
|
I did not want to suggest to put all bitstreams in one compressed file. That was only to test the decompression speed. |
|
Okay! Sorry to my misunderstood. |
|
for spioverjtag library where files are compressed with
known-to-work tool, zlib is old and stable so in ideal world
there should be near 0-maintenance. Of course we want
code with clean and compiler independent style.
Here's uzlib another maybe even simpler than miniz,
probably there are more project in this direction
https://github.com/pfalcon/uzlib
…On 11/17/21, Gwenhael Goavec-Merou ***@***.***> wrote:
Okay! Sorry to my misunderstood.
And thanks for your tests!
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#47 (comment)
|
|
I would say there are 3 ways:
To be honest, i would prefer option 2) since it is quite straightforward and not pain for user to use and build from the source on any platform. Hope it helps! |
|
I would go with good old zlib. That is already used in several projects, such as GHDL, GTKWave, Yosys... all of which are known to work on GNU/Linux, Windows and macOS. Actually, the difference between |
|
I don't have any experience with gzip libraries, but regarding packaging concerns on macOS both zlib and zlib-ng are well supported. |
|
Thanks to all! |
|
I have added the code to decompress gz files (works on linux, building for mingw is okay too).
macOS support has to be updated accordingly (currently hid and zlib aren't supported) Thanks all! |
|
Great!
I find it great if on-the-fly decompression would be also possible
with any normal bitstream, for example:
openFPGALoader -b ulx3s bitstream.bit.gz
…On 11/25/21, Gwenhael Goavec-Merou ***@***.***> wrote:
I have added the code to decompress gz files (works on linux, building for
mingw is okay too).
Currently:
- no bitstream are compressed (it's easy to compress manually for
validation), but new bitstreams will in this format
- it's limited to intel/altera (these bitstreams can't be internally
compressed), for xilinx I don't know if it's required since these are
already compressed by the tool.
- a fallback is present: if .gz file is not found openFPGALoader will search
for same file withouth .gz extension
macOS support has to be updated accordingly (currently hid and zlib aren't
supported)
Thanks all!
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#47 (comment)
|
|
In fact it's not really limited to spiOverJtag with altera: do the job. |
|
yes yes this works for "master" branch.
right, when .gz is found at the end, small
parser can check is there .bit .sof and similar
The .gz is really great because it's so efficient
at compressing bitstreams, bitstream distributions
will be very light using it
Thanx very much for great support
…On 11/25/21, Gwenhael Goavec-Merou ***@***.***> wrote:
In fact it's not really limited to spiOverJtag with altera:
```
openFPGALoader -b ulx3s bitstream.bit.gz --file-type bit
```
do the job.
Have to modify the file type detection to check if extension is gzip or gz
and to take into account second extension
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#47 (comment)
|
|
it's fixed on master branch. Now gz extension is detected, real type too. |
|
I can confirm on master branch that gzip -9 compressed bitstream works with
on-the-fly decompression:
openFPGALoader -b ulx3s bitstream.bit.gz
…On Fri, Nov 26, 2021 at 7:38 AM Gwenhael Goavec-Merou < ***@***.***> wrote:
it's fixed on master branch. Now gz extension is detected, real type too.
I think I need to simplify a bit. This detection is done twice: in
device.cpp and in configBitstreamParser.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#47 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AC4K2RZSSRBGMAA6BGIZA33UN4TQFANCNFSM4QUO3HVA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
|
Great to know that! |
HI!
I suggest gzip on-the-fly decompression to upload bitstreams.
Especially it's good for spi-over-jtag bitstreams database
each one from 2MB becomes only 3.3KB with gzip -9
The text was updated successfully, but these errors were encountered: