Adding ARM builds (2nd round) #4457

Closed
wants to merge 19 commits into
from

Conversation

8 participants
@NicoHood
Contributor

NicoHood commented Jan 20, 2016

Originally based on #3549

See the link above for additional recompile instructions of the different components and also the work of @ShorTie8 https://github.com/ShorTie8/Arduino_IDE

Take me to Download!

Just see the latest readme here which should point to the current downloads:
https://github.com/NicoHood/Arduino

Attention

This PR does include precompiled binaries! You should remove those before you merge this PR. I could do this on a different branch, if you are willing to merge this PR. But for now we keep this PR as visual reference of the changes I made to far to build.xml.

Adding precompiled binaries can be avoided if the arduino company uploads those files on their folders and also recompile new arm builds, such as the arduino-builder arm build (which needs a fix for now).

Build instructions

This PR can be build with an arm device or any x64 linux:

#x64
ant clean build dist -Dplatform=arm

#arm
ant clean build

To create new checksums use this:

shasum ./arduino-builder-arm-1.3.6.tar.bz2 | awk '{ print $1 }' > ./arduino-builder-arm-1.3.6.tar.bz2.sha

Compiling on arm with the binaries in place takes about 5~10min on a raspi2 which is acceptable since you do this once a day as dev.

Please note that crosscompiling the IDE is possible, since the binaries are precompiled on arm. The arm binaries are not yet all crosscompile able. Also it is not worth to add this option, since we only need to do that once (except for arduino-builder).

Bugs

Edit: Fixed! See below

The current build will not start! This happens with crosscompile and normal compilation.

arduino-1.6.8 $ ./arduino
Picked up JAVA_TOOL_OPTIONS: 
Tool name : avrdude 6.0.1-arduino5
     sys : i386-apple-darwin11 
     sys : x86_64-linux-gnu 
     sys : i686-linux-gnu 
     sys : i686-mingw32 
 seems to have no downloadable contributions for your operating system, but it is installed in
/home/pi/hackallthethings/arduino-1.6.8/hardware/tools/avr
Tool name : avr-gcc 4.8.1-arduino5
     sys : i386-apple-darwin11 
     sys : x86_64-linux-gnu 
     sys : i686-linux-gnu 
     sys : i686-mingw32 
 seems to have no downloadable contributions for your operating system, but it is installed in
/home/pi/hackallthethings/arduino-1.6.8/hardware/tools/avr
java.lang.NullPointerException
    at processing.app.BaseNoGui.createToolPreferences(BaseNoGui.java:851)
    at processing.app.BaseNoGui.initPackages(BaseNoGui.java:609)
    at processing.app.Base.<init>(Base.java:295)
    at processing.app.Base.guardedMain(Base.java:222)
    at processing.app.Base.main(Base.java:136)

This bug happens here:
https://github.com/NicoHood/Arduino/blob/master/arduino-core/src/cc/arduino/contributions/packages/ContributionsIndexer.java#L273

And might or might not be related to this:
6398cde

Which looks to me like it is a bug from the commit above:

public DownloadableContribution getDownloadableContribution(Platform platform) {

This commit caused the error, I reverted the commit:
6398cde

cc @cmaglie @facchinm @CRImier @imihalkovic @TXCodeDancer @Testato

NicoHood referenced this pull request Jan 20, 2016

Added 'arm' architecture support in Boards Manager
This is a necessary step to fully support ARM builds of the Arduino IDE

See #3549
@ShorTie8

This comment has been minimized.

Show comment
Hide comment
@ShorTie8

ShorTie8 Jan 24, 2016

Ya, you broke it when you updated to 1.6.8, lol.
Maybe try syncing your git ??

Ya, you broke it when you updated to 1.6.8, lol.
Maybe try syncing your git ??

@NicoHood

This comment has been minimized.

Show comment
Hide comment
@NicoHood

NicoHood Jan 24, 2016

Contributor

It has to do something with this commit:
6398cde

I think we can easily fix this one. I just dont know how, but @cmaglie knows how this works. Then I get update the git again and it should compile then without any additional steps.

Contributor

NicoHood commented Jan 24, 2016

It has to do something with this commit:
6398cde

I think we can easily fix this one. I just dont know how, but @cmaglie knows how this works. Then I get update the git again and it should compile then without any additional steps.

@ShorTie8

This comment has been minimized.

Show comment
Hide comment
@ShorTie8

ShorTie8 Jan 26, 2016

Yup, If you undo that patch, all is well.
I tried adding some stuff into package_index.json, but then it fails the sig check.
Which is some sort of funky file, so I can't make a new sig file.
It seems like there new motto is, DEATH to open source !!
ffissore was/is a big advocate of open source, the way the IDE is getting locked down with all those stupid exceptions and stuff, might be a cause for his moving on, but that is just MHO !!!!

They even make a hidden directory to hide the package_index.json and sig file

 locate package_index.json
/home/pi/.arduino15/package_index.json
/home/pi/.arduino15/package_index.json.sig
/usr/local/src/Arduino_IDE/package_index.json.diff

ps: @NicoHood your github could still use updating, there are no 'git tags' after 1.6.5-r5 .. :/~

Yup, If you undo that patch, all is well.
I tried adding some stuff into package_index.json, but then it fails the sig check.
Which is some sort of funky file, so I can't make a new sig file.
It seems like there new motto is, DEATH to open source !!
ffissore was/is a big advocate of open source, the way the IDE is getting locked down with all those stupid exceptions and stuff, might be a cause for his moving on, but that is just MHO !!!!

They even make a hidden directory to hide the package_index.json and sig file

 locate package_index.json
/home/pi/.arduino15/package_index.json
/home/pi/.arduino15/package_index.json.sig
/usr/local/src/Arduino_IDE/package_index.json.diff

ps: @NicoHood your github could still use updating, there are no 'git tags' after 1.6.5-r5 .. :/~

@Testato

This comment has been minimized.

Show comment
Hide comment
@Testato

Testato Jan 26, 2016

+1
Il 26/gen/2016 08:52, "ShorTie8" notifications@github.com ha scritto:

Yup, If you undo that patch, all is well.
I tried adding some stuff into package_index.json, but then it fails the
sig check.
Which is some sort of funky file, so I can't make a new sig file.
It seems like there new motto is, DEATH to open source !!
ffissore was/is a big advocate of open source, the way the IDE is getting
locked down with all those stupid exceptions and stuff, might be a cause
for his moving on, but that is just MHO !!!!


Reply to this email directly or view it on GitHub
#4457 (comment).

Testato commented Jan 26, 2016

+1
Il 26/gen/2016 08:52, "ShorTie8" notifications@github.com ha scritto:

Yup, If you undo that patch, all is well.
I tried adding some stuff into package_index.json, but then it fails the
sig check.
Which is some sort of funky file, so I can't make a new sig file.
It seems like there new motto is, DEATH to open source !!
ffissore was/is a big advocate of open source, the way the IDE is getting
locked down with all those stupid exceptions and stuff, might be a cause
for his moving on, but that is just MHO !!!!


Reply to this email directly or view it on GitHub
#4457 (comment).

@NicoHood

This comment has been minimized.

Show comment
Hide comment
@NicoHood

NicoHood Jan 26, 2016

Contributor

IDE is now fixed and a download is provided. Please test and report if it works on your device and which device you used.

The current master is broken, due to the lack of a new arduino-builder arm version. But I tagged the working version, so you can always go back to this and recompile it yourself.
https://github.com/NicoHood/Arduino

Contributor

NicoHood commented Jan 26, 2016

IDE is now fixed and a download is provided. Please test and report if it works on your device and which device you used.

The current master is broken, due to the lack of a new arduino-builder arm version. But I tagged the working version, so you can always go back to this and recompile it yourself.
https://github.com/NicoHood/Arduino

@cmaglie

This comment has been minimized.

Show comment
Hide comment
@cmaglie

cmaglie Jan 26, 2016

Member

The patch on the HostDownloadable class has been made to allow updates of the cores from the Boards Manager. As you have noticed the packages_index.json still misses the tools section with the arm flavour of the tools, you can add it by yourself and rename package_index.json to test_package_index.json for testing (adding the test_ prefix will skip the signature check).

Without this patch the next time you try to update the AVR core, or install a non-AVR core, through the Board Manager the IDE will download the tools for x86 and consequently fail: that's the reason why I added the patch, not to lock someone out.

ffissore was/is a big advocate of open source, the way the IDE is getting locked down with all those stupid exceptions and stuff, might be a cause for his moving on, but that is just MHO !!!!

@ShorTie8
I can understand your frustration but, as you can see from the git history, the signature check has been added by ffissore himself, a long time ago, and for a very good reason. I can assure you that this is not the cause of his moving and I would like to ask you (and anyone else) to not make FUD out of personal choices that someone may have made for a number of different reasons that are out of your/our knowledge.

Thanks.

Member

cmaglie commented Jan 26, 2016

The patch on the HostDownloadable class has been made to allow updates of the cores from the Boards Manager. As you have noticed the packages_index.json still misses the tools section with the arm flavour of the tools, you can add it by yourself and rename package_index.json to test_package_index.json for testing (adding the test_ prefix will skip the signature check).

Without this patch the next time you try to update the AVR core, or install a non-AVR core, through the Board Manager the IDE will download the tools for x86 and consequently fail: that's the reason why I added the patch, not to lock someone out.

ffissore was/is a big advocate of open source, the way the IDE is getting locked down with all those stupid exceptions and stuff, might be a cause for his moving on, but that is just MHO !!!!

@ShorTie8
I can understand your frustration but, as you can see from the git history, the signature check has been added by ffissore himself, a long time ago, and for a very good reason. I can assure you that this is not the cause of his moving and I would like to ask you (and anyone else) to not make FUD out of personal choices that someone may have made for a number of different reasons that are out of your/our knowledge.

Thanks.

@cmaglie

This comment has been minimized.

Show comment
Hide comment
@cmaglie

cmaglie Jan 26, 2016

Member

Adding precompiled binaries can be avoided if the arduino company uploads those files on their folders and also recompile new arm builds, such as the arduino-builder arm build (which needs a fix for now).

@NicoHood
We are going to setup an ARM builder on our CI system, we'll provide an ARM build shortly (at least for the arduino-builder it should be easy).

Member

cmaglie commented Jan 26, 2016

Adding precompiled binaries can be avoided if the arduino company uploads those files on their folders and also recompile new arm builds, such as the arduino-builder arm build (which needs a fix for now).

@NicoHood
We are going to setup an ARM builder on our CI system, we'll provide an ARM build shortly (at least for the arduino-builder it should be easy).

@NicoHood

This comment has been minimized.

Show comment
Hide comment
@NicoHood

NicoHood Jan 26, 2016

Contributor

Oh I see now. Thx for clarifying @cmaglie

So what about arduino hosting the binaries? Then I can add the download link instead and remove the binaries from the git. This would make things simpler.

If the arm arduino-builder (with the path fix) is setup and the other files are hosted it should be easy to maintain this. It would be almost ready to merge. We just have to keep track of the changes and apply those additions for arm too. But most of the work is done already.

Contributor

NicoHood commented Jan 26, 2016

Oh I see now. Thx for clarifying @cmaglie

So what about arduino hosting the binaries? Then I can add the download link instead and remove the binaries from the git. This would make things simpler.

If the arm arduino-builder (with the path fix) is setup and the other files are hosted it should be easy to maintain this. It would be almost ready to merge. We just have to keep track of the changes and apply those additions for arm too. But most of the work is done already.

@cmaglie

This comment has been minimized.

Show comment
Hide comment
@cmaglie

cmaglie Jan 26, 2016

Member

So what about arduino hosting the binaries?

Sure, but we can't host untrusted binaries, we need to reproduce the build here. Let's start with arduino-builder, I'll give it a spin tomorrow, now I should go AFK for a while.

Member

cmaglie commented Jan 26, 2016

So what about arduino hosting the binaries?

Sure, but we can't host untrusted binaries, we need to reproduce the build here. Let's start with arduino-builder, I'll give it a spin tomorrow, now I should go AFK for a while.

@NicoHood

This comment has been minimized.

Show comment
Hide comment
@NicoHood

NicoHood Jan 26, 2016

Contributor

Sure, thats true.
You can use the steps I used above (except for arduino builder, you know this one better). Then the checksum should be the same. Keep in mind that most arm stuff has to be compile on an arm device. RaspberryPi2 works perfect for me and in reasonable time (toolchain takes long and has a missing download mirror atm for all systems).

Contributor

NicoHood commented Jan 26, 2016

Sure, thats true.
You can use the steps I used above (except for arduino builder, you know this one better). Then the checksum should be the same. Keep in mind that most arm stuff has to be compile on an arm device. RaspberryPi2 works perfect for me and in reasonable time (toolchain takes long and has a missing download mirror atm for all systems).

@ShorTie8

This comment has been minimized.

Show comment
Hide comment
@ShorTie8

ShorTie8 Jan 27, 2016

fyi: My script always builds the latest git versions, but may tar them up as a different version so the build.xml is satisfied. And I don't hide what it is doing, I'm nice and it tells ya in the push to be.

Thankz @NicoHood for updating and fixing github

ps @cmaglie it's my computer not yours, why do write files that I can not normally see ??
Like your trying to hide something that peeps can't see it !!!!!!!!
Thought this was open source and not find me if you can !!!!!
And if you look at what ffissore said a couple of months ago, it sure seems to me that some thing turn his stomach quickly to make him move on, but that is just MF`nHO.

So how do you make the sig file ???

fyi: My script always builds the latest git versions, but may tar them up as a different version so the build.xml is satisfied. And I don't hide what it is doing, I'm nice and it tells ya in the push to be.

Thankz @NicoHood for updating and fixing github

ps @cmaglie it's my computer not yours, why do write files that I can not normally see ??
Like your trying to hide something that peeps can't see it !!!!!!!!
Thought this was open source and not find me if you can !!!!!
And if you look at what ffissore said a couple of months ago, it sure seems to me that some thing turn his stomach quickly to make him move on, but that is just MF`nHO.

So how do you make the sig file ???

@facchinm

This comment has been minimized.

Show comment
Hide comment
@facchinm

facchinm Jan 27, 2016

Member

@ShorTie8, to make the sig file you only need echoshasum $filename> $filename.sig.
But, please, stop acting like we hid something and stop talking about people you don't know. Everything in the build system is crystal clear, the utility to generate the signature is open source and the code to check if the downloaded file is valid is open source too.
If you are worried about your security you only need to read the code 😉

Member

facchinm commented Jan 27, 2016

@ShorTie8, to make the sig file you only need echoshasum $filename> $filename.sig.
But, please, stop acting like we hid something and stop talking about people you don't know. Everything in the build system is crystal clear, the utility to generate the signature is open source and the code to check if the downloaded file is valid is open source too.
If you are worried about your security you only need to read the code 😉

@ShorTie8

This comment has been minimized.

Show comment
Hide comment
@ShorTie8

ShorTie8 Jan 27, 2016

@facchinm oh really ??

pi Arduino_IDE # cat /home/pi/.arduino15/package_index.json.sig
▒V▒Y▒
        $▒k)B▒Yc▒H2▒▒8▒bEv▒=▒s▒▒W▒▒m▒zA▒y▒J*▒K▒▒▒:▒Ou,kT▒▒(LP▒▒▒K00i ▒▒Tn[▒?▒▒)▒[u▒}▒▒▒:]▒▒▒▒▒G▒k▒K▒t▒▒
▒▒      c▒#▒ǯI
              ▒Dhv%▒e▒▒*ߵu▒2 ▒▒/▒▒H▒a▒▒▒▒
㰑0P▒▒3▒▒▒?▒
            ▒P2▒}+▒▒▒▒.j▒▒Y\▒P▒▒▒)▒ů▒O▒▒:&▒▒z▒▒▒i▒▒▒▒o▒\▒▒l▒▒
@▒+6▒▒M▒▒
         ▒0▒▒j▒▒I▒▒▒▒▒▒w▒▒▒6▒(▒▒V▒el▒▒[▒▒Rn▒▒▒▒▒tn'▒▒▒(z▒▒K0▒▒▒▒▒▒XՃB▒Z▒M▒Ta▒▒P˿m▒)▒▒▒˥jr▒PMj▒▒T▒_}▒▒▒▒c(▒d▒J▒▒6▒▒g4▒(▒7ܢ▒▒▒▒N▒.6▒t٘B▒▒▒2▒▒▒▒▒-QfV▒R▒▒}d▒▒▒X▒M▒▒▒;▒▒▒▒▒H▒4q▒C▒▒Z▒▒▒▒▒w   &▒▒$Ǐq▒Q▒▒▒▒▒▒s▒@▒k▒r ▒▒▒▒t▒
▒/▒▒֎▒▒h▒▒~ڀX▒\▒r▒PuTTYpi Arduino_IDE # PuTTY

@facchinm oh really ??

pi Arduino_IDE # cat /home/pi/.arduino15/package_index.json.sig
▒V▒Y▒
        $▒k)B▒Yc▒H2▒▒8▒bEv▒=▒s▒▒W▒▒m▒zA▒y▒J*▒K▒▒▒:▒Ou,kT▒▒(LP▒▒▒K00i ▒▒Tn[▒?▒▒)▒[u▒}▒▒▒:]▒▒▒▒▒G▒k▒K▒t▒▒
▒▒      c▒#▒ǯI
              ▒Dhv%▒e▒▒*ߵu▒2 ▒▒/▒▒H▒a▒▒▒▒
㰑0P▒▒3▒▒▒?▒
            ▒P2▒}+▒▒▒▒.j▒▒Y\▒P▒▒▒)▒ů▒O▒▒:&▒▒z▒▒▒i▒▒▒▒o▒\▒▒l▒▒
@▒+6▒▒M▒▒
         ▒0▒▒j▒▒I▒▒▒▒▒▒w▒▒▒6▒(▒▒V▒el▒▒[▒▒Rn▒▒▒▒▒tn'▒▒▒(z▒▒K0▒▒▒▒▒▒XՃB▒Z▒M▒Ta▒▒P˿m▒)▒▒▒˥jr▒PMj▒▒T▒_}▒▒▒▒c(▒d▒J▒▒6▒▒g4▒(▒7ܢ▒▒▒▒N▒.6▒t٘B▒▒▒2▒▒▒▒▒-QfV▒R▒▒}d▒▒▒X▒M▒▒▒;▒▒▒▒▒H▒4q▒C▒▒Z▒▒▒▒▒w   &▒▒$Ǐq▒Q▒▒▒▒▒▒s▒@▒k▒r ▒▒▒▒t▒
▒/▒▒֎▒▒h▒▒~ڀX▒\▒r▒PuTTYpi Arduino_IDE # PuTTY
@ShorTie8

This comment has been minimized.

Show comment
Hide comment
@ShorTie8

ShorTie8 Jan 27, 2016

@cmaglie Actually it seems I think Board Manager is smart enough not to offer any platform specific downloads for unknown os types. Or so it seems because I was trying to add the sam stuff for yucks and couldn't find it.

But truely not sure, and not worth my time to figure further, I'll just hack the build.xml to include it if peeps want it and see how it goes .. :/~

So I got the IDE and toolchain-avr of course, arduino-builder, astyle, bossac, Coan, ctags, OpenOCD and listSerialPortsC all being compiled and/or options to. Some of witch I don't know what they are specifically for or where they might go .. :/~

What other os specific things need compiling ??

@cmaglie Actually it seems I think Board Manager is smart enough not to offer any platform specific downloads for unknown os types. Or so it seems because I was trying to add the sam stuff for yucks and couldn't find it.

But truely not sure, and not worth my time to figure further, I'll just hack the build.xml to include it if peeps want it and see how it goes .. :/~

So I got the IDE and toolchain-avr of course, arduino-builder, astyle, bossac, Coan, ctags, OpenOCD and listSerialPortsC all being compiled and/or options to. Some of witch I don't know what they are specifically for or where they might go .. :/~

What other os specific things need compiling ??

@cmaglie

This comment has been minimized.

Show comment
Hide comment
@cmaglie

cmaglie Jan 27, 2016

Member

@ShorTie8 do you want to express your disappontment? that's fine, I'm just asking to not talk on behalf of someone else, especially if that someone else is unaware of that and what are you saying is just your opinion.

to make the sig file you only need echo shasum $filename > $filename.sig`

shasum is needed to make the .sha files needed on build.xml.

To make the .sig file we use gpg and a private key to sign the package_index.json file. This is a common way to use strong crypto to validate downloaded package and protect against tampering.
Just for the note:

  • the same approach is adopted by the major linux distributions (and the reason is clear, literature is full of cases when a single file compromised in an FTP server lead to thousand of user's PC infected).
  • we do the check above on our package index, but anyone is free to use his own package_xxxxx_index.json, like a lot of 3rd party producer, unaffiliated with arduino, does (see https://github.com/arduino/Arduino/wiki/Unofficial-list-of-3rd-party-boards-support-urls)

Hope this clarifies the reasons behind the .sig file.

Member

cmaglie commented Jan 27, 2016

@ShorTie8 do you want to express your disappontment? that's fine, I'm just asking to not talk on behalf of someone else, especially if that someone else is unaware of that and what are you saying is just your opinion.

to make the sig file you only need echo shasum $filename > $filename.sig`

shasum is needed to make the .sha files needed on build.xml.

To make the .sig file we use gpg and a private key to sign the package_index.json file. This is a common way to use strong crypto to validate downloaded package and protect against tampering.
Just for the note:

  • the same approach is adopted by the major linux distributions (and the reason is clear, literature is full of cases when a single file compromised in an FTP server lead to thousand of user's PC infected).
  • we do the check above on our package index, but anyone is free to use his own package_xxxxx_index.json, like a lot of 3rd party producer, unaffiliated with arduino, does (see https://github.com/arduino/Arduino/wiki/Unofficial-list-of-3rd-party-boards-support-urls)

Hope this clarifies the reasons behind the .sig file.

@cmaglie

This comment has been minimized.

Show comment
Hide comment
@cmaglie

cmaglie Jan 27, 2016

Member

@cmaglie Actually it seems I think Board Manager is smart enough not to offer any platform specific downloads for unknown os types. Or so it seems because I was trying to add the sam stuff for yucks and couldn't find it.

Yes that's the reason why I added the extra case for ARM, once the builds are in place everything should go smooth ;-)

Member

cmaglie commented Jan 27, 2016

@cmaglie Actually it seems I think Board Manager is smart enough not to offer any platform specific downloads for unknown os types. Or so it seems because I was trying to add the sam stuff for yucks and couldn't find it.

Yes that's the reason why I added the extra case for ARM, once the builds are in place everything should go smooth ;-)

@NicoHood

This comment has been minimized.

Show comment
Hide comment
@NicoHood

NicoHood Jan 27, 2016

Contributor

Could we please go back to topic and just add ARM builds? You can discuss on google I think.
It would really help to focus now, since we are already done. I do not want this to fail just because some users have personal problems with the Arduino IDE.

So I guess we will wait until Christian fixes the arduino-builder packages. I am very glad that you (all) help supporting the ARM builds soon :)

Contributor

NicoHood commented Jan 27, 2016

Could we please go back to topic and just add ARM builds? You can discuss on google I think.
It would really help to focus now, since we are already done. I do not want this to fail just because some users have personal problems with the Arduino IDE.

So I guess we will wait until Christian fixes the arduino-builder packages. I am very glad that you (all) help supporting the ARM builds soon :)

@ShorTie8

This comment has been minimized.

Show comment
Hide comment
@ShorTie8

ShorTie8 Jan 27, 2016

Sorry @NicoHood i'll hush. After 1 last thing.
fwi: @cmaglie @facchinm I've been basically dicking around with this shit for 2 years providing peeps a way to use your stupid program on ARM's because your to lazy to provide it, and I really never use it or don't even need it !!!!!
And I think if you setup Java right, the old java compiler might just work fine,

<property name="java_home" value="${env.JAVA_HOME}" />

is leaving a lot to assumption.
But I guess I've at least learned a few things, like 'exceptions', so maybe it's worth while.
So I'm not expecting any thing soon, except maybe more exceptions !!
Or some other F`n way to mess it up for me/us.

Sorry @NicoHood i'll hush. After 1 last thing.
fwi: @cmaglie @facchinm I've been basically dicking around with this shit for 2 years providing peeps a way to use your stupid program on ARM's because your to lazy to provide it, and I really never use it or don't even need it !!!!!
And I think if you setup Java right, the old java compiler might just work fine,

<property name="java_home" value="${env.JAVA_HOME}" />

is leaving a lot to assumption.
But I guess I've at least learned a few things, like 'exceptions', so maybe it's worth while.
So I'm not expecting any thing soon, except maybe more exceptions !!
Or some other F`n way to mess it up for me/us.

@cmaglie

This comment has been minimized.

Show comment
Hide comment
@cmaglie

cmaglie Jan 28, 2016

Member

Ok good news so far, I've been able to reproduce all the builds (didn't tested everything yet BTW).

Here a recap:

What how to build repo
avr-gcc, avrdude arch.arm.build.bash provided by @CRImier https://github.com/arduino/toolchain-avr
coan configure & make downloaded coan-5.2.tar.gz from sf.net
ctags configure & make https://github.com/arduino/ctags
libastyle arm.bash provided by @CRImier https://github.com/arduino/astyle
OpenOCD arch.arm.build.bash https://github.com/arduino/OpenOCD-build-script
bossac make bin/bossac https://github.com/shumatech/BOSSA/tree/arduino
liblistserial compile_linux.sh from @NicoHood https://github.com/arduino/listSerialPortsC

Note that it's still missing arm-none-eabi-gcc, this is much complex to build and we need exactly the version 4.8.3-2014q1.

I've uploaded all the compiled binaries to the server together with an updated test package index that can be found here: http://downloads.arduino.cc/packages/test_package_arm_index.json
(the diff is here: https://gist.github.com/cmaglie/993679418eca1426f906 )
@NicoHood you can try to remove the binaries from your repo now I guess?
note: the arm-none-eabi-gcc is a fake entry to allow tests to run, will be replaced by the real version when available

The updates to arduino-builder to support ARM are in the following branch:
https://github.com/arduino/arduino-builder/compare/arm?expand=1

there are basically two changes:

  • I've added a case for "linux-arm" to support tools for armhf
  • changed the download url of package_index to allow testing with the new tools for armhf

so far I'm able to run the tests and much of them are passing, here the list of the remaining failures:

$ go test -timeout 60m -v ./src/arduino.cc/builder/test/... 2>&1 | grep FAIL
--- FAIL: TestBuilderBridgeSAM (0.42s)
--- FAIL: TestIncludesToIncludeFoldersDuplicateLibs2 (0.57s)
--- FAIL: TestPrototypesAdderIncludeBetweenMultilineComment (0.41s)
--- FAIL: TestPrototypesAdderSketchWithIfDef2SAM (0.40s)
--- FAIL: TestTime (0.00s)
--- FAIL: TestTryBuild022 (1.28s)
FAIL
FAIL    arduino.cc/builder/test 1222.002s

I think that are due to the missing arm-none-eabi-gcc (except TestTime maybe, something to check).

Member

cmaglie commented Jan 28, 2016

Ok good news so far, I've been able to reproduce all the builds (didn't tested everything yet BTW).

Here a recap:

What how to build repo
avr-gcc, avrdude arch.arm.build.bash provided by @CRImier https://github.com/arduino/toolchain-avr
coan configure & make downloaded coan-5.2.tar.gz from sf.net
ctags configure & make https://github.com/arduino/ctags
libastyle arm.bash provided by @CRImier https://github.com/arduino/astyle
OpenOCD arch.arm.build.bash https://github.com/arduino/OpenOCD-build-script
bossac make bin/bossac https://github.com/shumatech/BOSSA/tree/arduino
liblistserial compile_linux.sh from @NicoHood https://github.com/arduino/listSerialPortsC

Note that it's still missing arm-none-eabi-gcc, this is much complex to build and we need exactly the version 4.8.3-2014q1.

I've uploaded all the compiled binaries to the server together with an updated test package index that can be found here: http://downloads.arduino.cc/packages/test_package_arm_index.json
(the diff is here: https://gist.github.com/cmaglie/993679418eca1426f906 )
@NicoHood you can try to remove the binaries from your repo now I guess?
note: the arm-none-eabi-gcc is a fake entry to allow tests to run, will be replaced by the real version when available

The updates to arduino-builder to support ARM are in the following branch:
https://github.com/arduino/arduino-builder/compare/arm?expand=1

there are basically two changes:

  • I've added a case for "linux-arm" to support tools for armhf
  • changed the download url of package_index to allow testing with the new tools for armhf

so far I'm able to run the tests and much of them are passing, here the list of the remaining failures:

$ go test -timeout 60m -v ./src/arduino.cc/builder/test/... 2>&1 | grep FAIL
--- FAIL: TestBuilderBridgeSAM (0.42s)
--- FAIL: TestIncludesToIncludeFoldersDuplicateLibs2 (0.57s)
--- FAIL: TestPrototypesAdderIncludeBetweenMultilineComment (0.41s)
--- FAIL: TestPrototypesAdderSketchWithIfDef2SAM (0.40s)
--- FAIL: TestTime (0.00s)
--- FAIL: TestTryBuild022 (1.28s)
FAIL
FAIL    arduino.cc/builder/test 1222.002s

I think that are due to the missing arm-none-eabi-gcc (except TestTime maybe, something to check).

@PaulStoffregen

This comment has been minimized.

Show comment
Hide comment
@PaulStoffregen

PaulStoffregen Jan 28, 2016

Collaborator

Note that it's still missing arm-none-eabi-gcc, this is much complex to build and we need exactly the version 4.8.3-2014q1.

I'll look into this part...

Collaborator

PaulStoffregen commented Jan 28, 2016

Note that it's still missing arm-none-eabi-gcc, this is much complex to build and we need exactly the version 4.8.3-2014q1.

I'll look into this part...

@wjwieland

This comment has been minimized.

Show comment
Hide comment
@wjwieland

wjwieland Jan 29, 2016

Just downloaded and installed arduino-1.6.8 on my RPI2. Fired right up. This is in contrast to packaged version installed via apt-get, which would not run at all. Very nice!

Just downloaded and installed arduino-1.6.8 on my RPI2. Fired right up. This is in contrast to packaged version installed via apt-get, which would not run at all. Very nice!

@cmaglie cmaglie referenced this pull request in arduino/arduino-builder Jan 29, 2016

Closed

arm-none-eabi-gcc build for armhf #105

@PaulStoffregen

This comment has been minimized.

Show comment
Hide comment
@PaulStoffregen

PaulStoffregen Jan 29, 2016

Collaborator

I'm making good progress on the 2014q1 toolchain build. Details here: arduino/arduino-builder#105 (comment)

Collaborator

PaulStoffregen commented Jan 29, 2016

I'm making good progress on the 2014q1 toolchain build. Details here: arduino/arduino-builder#105 (comment)

@NicoHood

This comment has been minimized.

Show comment
Hide comment
@NicoHood

NicoHood Jan 30, 2016

Contributor

Thx for uploading the binaries. However the checksums did not match the checksums that I compiled myself. Maybe you had a different compiler version.

However I've updated the checksums, reverted the patch (for the json file) and updated the latest git. I also remove the binaries now.

@cmaglie The package_index.json does not seem to work for me. I still get this error, that no proper versions were found:

   83  cd arduino-1.6.8/
   85  cd dist/
   86  wget http://downloads.arduino.cc/packages/test_package_arm_index.json
   89  mv test_package_arm_index.json test_package_index.json 
   90  rm package_index.json

It seems to totally ignore the file (even if I delete it), so I do not know where it is searching. Or do I have to add this before compiling the IDE? Also could you please add the .sig file too (without the arm in the name, if that matters)?

Edit: I had to remove the file from .arduino15/package_index.json. However the test prefix does not skip the signing.

It also looks like this file totally differs from the current git: http://downloads.arduino.cc/packages/test_package_arm_index.json Also I could not apply the patch.

Then we need to merge the commit together (to remove the binaries from the history), test the IDE (on different systems) and also remove the additional readme that I added.

Contributor

NicoHood commented Jan 30, 2016

Thx for uploading the binaries. However the checksums did not match the checksums that I compiled myself. Maybe you had a different compiler version.

However I've updated the checksums, reverted the patch (for the json file) and updated the latest git. I also remove the binaries now.

@cmaglie The package_index.json does not seem to work for me. I still get this error, that no proper versions were found:

   83  cd arduino-1.6.8/
   85  cd dist/
   86  wget http://downloads.arduino.cc/packages/test_package_arm_index.json
   89  mv test_package_arm_index.json test_package_index.json 
   90  rm package_index.json

It seems to totally ignore the file (even if I delete it), so I do not know where it is searching. Or do I have to add this before compiling the IDE? Also could you please add the .sig file too (without the arm in the name, if that matters)?

Edit: I had to remove the file from .arduino15/package_index.json. However the test prefix does not skip the signing.

It also looks like this file totally differs from the current git: http://downloads.arduino.cc/packages/test_package_arm_index.json Also I could not apply the patch.

Then we need to merge the commit together (to remove the binaries from the history), test the IDE (on different systems) and also remove the additional readme that I added.

cmaglie added a commit to cmaglie/Arduino that referenced this pull request Feb 1, 2016

@cmaglie cmaglie self-assigned this Feb 1, 2016

@cmaglie

This comment has been minimized.

Show comment
Hide comment
@cmaglie

cmaglie Feb 1, 2016

Member

@NicoHood
I've rebased and squashed your commits in this branch:
https://github.com/arduino/Arduino/compare/master...cmaglie:arm-2?expand=1

This lead to a very clean history as you described, and now I'm able to build the IDE on the raspberry and run it successfully.

The package_index.json does not seem to work for me.

Yes, sorry, I forget to tell you that you can set the java property PACKAGE_INDEX_URL to override the default URL used by the IDE to download the package index, BTW you're right this won't bypass the signature check so I added a this possibility here 0e471bc

After that you can overrides the package_index.json in this way: c35250d (note that this commit will be removed before merging the branch upstream).

Now I'm looking into ArduinoBot to allow it build also the ARM version. We are very close!

Member

cmaglie commented Feb 1, 2016

@NicoHood
I've rebased and squashed your commits in this branch:
https://github.com/arduino/Arduino/compare/master...cmaglie:arm-2?expand=1

This lead to a very clean history as you described, and now I'm able to build the IDE on the raspberry and run it successfully.

The package_index.json does not seem to work for me.

Yes, sorry, I forget to tell you that you can set the java property PACKAGE_INDEX_URL to override the default URL used by the IDE to download the package index, BTW you're right this won't bypass the signature check so I added a this possibility here 0e471bc

After that you can overrides the package_index.json in this way: c35250d (note that this commit will be removed before merging the branch upstream).

Now I'm looking into ArduinoBot to allow it build also the ARM version. We are very close!

@NicoHood

This comment has been minimized.

Show comment
Hide comment
@NicoHood

NicoHood Feb 2, 2016

Contributor

Great work done!
So I guess after ArduinoBot can also build ARM builds we are done and also dont need to maintain 3rd party arm builds? This would be totally fantastic!

I am going to test this later. We then could add a list of tested boards and check if all of them work properly. Also the diff itself is quite simple. It was a lot of work to get there, but actually its impressive that its not much to change. ;)

Contributor

NicoHood commented Feb 2, 2016

Great work done!
So I guess after ArduinoBot can also build ARM builds we are done and also dont need to maintain 3rd party arm builds? This would be totally fantastic!

I am going to test this later. We then could add a list of tested boards and check if all of them work properly. Also the diff itself is quite simple. It was a lot of work to get there, but actually its impressive that its not much to change. ;)

@PaulStoffregen

This comment has been minimized.

Show comment
Hide comment
@PaulStoffregen

PaulStoffregen Feb 2, 2016

Collaborator

About 10 hours are left until the (hopefully) final build of the toolchain for Due & Zero is complete.
arduino/arduino-builder#105

Collaborator

PaulStoffregen commented Feb 2, 2016

About 10 hours are left until the (hopefully) final build of the toolchain for Due & Zero is complete.
arduino/arduino-builder#105

@PaulStoffregen

This comment has been minimized.

Show comment
Hide comment
@PaulStoffregen

PaulStoffregen Feb 3, 2016

Collaborator

Finished building and checking the 2014q1 toolchain. :)
arduino/arduino-builder#105 (comment)

Collaborator

PaulStoffregen commented Feb 3, 2016

Finished building and checking the 2014q1 toolchain. :)
arduino/arduino-builder#105 (comment)

@cmaglie cmaglie referenced this pull request in arduino/arduino-builder Feb 5, 2016

Closed

run arduino-builder on ARM Linux box #76

@facchinm facchinm referenced this pull request Feb 8, 2016

Closed

Adding ARM builds #3549

cmaglie added a commit to cmaglie/Arduino that referenced this pull request Feb 9, 2016

cmaglie added a commit to cmaglie/Arduino that referenced this pull request Feb 24, 2016

@theanch

This comment has been minimized.

Show comment
Hide comment
@theanch

theanch Mar 20, 2016

Hi, I am trying to install the Arduino Two board on Raspberry but at compile time I getting the message "arm-none-eabi-g ++ exec format error" even with the 2014q1 toolchain by PaulStoffregen. Can someone help me?

theanch commented Mar 20, 2016

Hi, I am trying to install the Arduino Two board on Raspberry but at compile time I getting the message "arm-none-eabi-g ++ exec format error" even with the 2014q1 toolchain by PaulStoffregen. Can someone help me?

@PaulStoffregen

This comment has been minimized.

Show comment
Hide comment
@PaulStoffregen

PaulStoffregen Mar 20, 2016

Collaborator

Can someone help me?

Maybe, but what steps would I need to take to recreate this error? I don't even know which RPi you're using. I can't see your screen, or what things you've click or what commands you've typed.

I do know that toolchain runs fine on my Raspberry Pi. I'm absolutely confident if I try it again the way I know how, it'll work perfectly, as it did a few weeks ago. Can you understand my reluctance to spend time on this without clear info about exactly the steps you took?

Collaborator

PaulStoffregen commented Mar 20, 2016

Can someone help me?

Maybe, but what steps would I need to take to recreate this error? I don't even know which RPi you're using. I can't see your screen, or what things you've click or what commands you've typed.

I do know that toolchain runs fine on my Raspberry Pi. I'm absolutely confident if I try it again the way I know how, it'll work perfectly, as it did a few weeks ago. Can you understand my reluctance to spend time on this without clear info about exactly the steps you took?

@theanch

This comment has been minimized.

Show comment
Hide comment
@theanch

theanch Mar 20, 2016

Thanks for the reply. If you say it works then definitely something I may be wrong. Meanwhile, I have a Raspberry pi 3, and I'm using the build of NicoHood 1.6.8-rc. Everything ok for Arduino Uno / Mega. I have the need to program for an Arduino Due, in the Arduino IDE. I go to Tools-> Board Manager and add "Sam Arduino Board 1.6.7". Once completed I select the Arduino card Two and I try to compile I get error:
fork / exec /home/pi/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++: exec format error.
Now I try to replace "arm-none-eabi-gcc" in /home/pi/.arduino15/packages/arduino/tools/arm-none-eabi-gcc with your build for ARM.
But I have always the same error:
fork / exec /home/pi/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++: exec format error.

Definitely wrong in something .. and I apologize for my stupidity

theanch commented Mar 20, 2016

Thanks for the reply. If you say it works then definitely something I may be wrong. Meanwhile, I have a Raspberry pi 3, and I'm using the build of NicoHood 1.6.8-rc. Everything ok for Arduino Uno / Mega. I have the need to program for an Arduino Due, in the Arduino IDE. I go to Tools-> Board Manager and add "Sam Arduino Board 1.6.7". Once completed I select the Arduino card Two and I try to compile I get error:
fork / exec /home/pi/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++: exec format error.
Now I try to replace "arm-none-eabi-gcc" in /home/pi/.arduino15/packages/arduino/tools/arm-none-eabi-gcc with your build for ARM.
But I have always the same error:
fork / exec /home/pi/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++: exec format error.

Definitely wrong in something .. and I apologize for my stupidity

@PaulStoffregen

This comment has been minimized.

Show comment
Hide comment
@PaulStoffregen

PaulStoffregen Mar 20, 2016

Collaborator

On your RPi in a terminal, can you try these command, and post the result?

md5sum /home/pi/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++
file /home/pi/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++

When I run them here on my Raspberry Pi, with the pathname to that file, I see this:

paul@raspberrypi ~ $ md5sum gcc-arm-none-eabi-4_8-2014q1/bin/arm-none-eabi-g++
bfd196cbc3e85ec922d8d5d7fa8d5449  gcc-arm-none-eabi-4_8-2014q1/bin/arm-none-eabi-g++
paul@raspberrypi ~ $ file gcc-arm-none-eabi-4_8-2014q1/bin/arm-none-eabi-g++
gcc-arm-none-eabi-4_8-2014q1/bin/arm-none-eabi-g++: ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.26, BuildID[sha1]=0x539864a83fc520418c0686b9d66e9da1a2f5fd44, stripped
Collaborator

PaulStoffregen commented Mar 20, 2016

On your RPi in a terminal, can you try these command, and post the result?

md5sum /home/pi/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++
file /home/pi/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++

When I run them here on my Raspberry Pi, with the pathname to that file, I see this:

paul@raspberrypi ~ $ md5sum gcc-arm-none-eabi-4_8-2014q1/bin/arm-none-eabi-g++
bfd196cbc3e85ec922d8d5d7fa8d5449  gcc-arm-none-eabi-4_8-2014q1/bin/arm-none-eabi-g++
paul@raspberrypi ~ $ file gcc-arm-none-eabi-4_8-2014q1/bin/arm-none-eabi-g++
gcc-arm-none-eabi-4_8-2014q1/bin/arm-none-eabi-g++: ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.26, BuildID[sha1]=0x539864a83fc520418c0686b9d66e9da1a2f5fd44, stripped
@theanch

This comment has been minimized.

Show comment
Hide comment
@theanch

theanch Mar 20, 2016

I do not know what strange reason, but does not copy the files .. they do not matched the md5! Now it works beautifully! Thank you so much for your patience, I owe you a drink

theanch commented Mar 20, 2016

I do not know what strange reason, but does not copy the files .. they do not matched the md5! Now it works beautifully! Thank you so much for your patience, I owe you a drink

cmaglie added a commit to cmaglie/Arduino that referenced this pull request Apr 8, 2016

@cmaglie

This comment has been minimized.

Show comment
Hide comment
@cmaglie

cmaglie Apr 8, 2016

Member

Supersed by #4517

Member

cmaglie commented Apr 8, 2016

Supersed by #4517

@cmaglie cmaglie closed this Apr 8, 2016

@cmaglie cmaglie added this to the Release 1.6.9 milestone Apr 8, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment