Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding ARM builds (2nd round) #4457

Closed
wants to merge 19 commits into from
Closed

Adding ARM builds (2nd round) #4457

wants to merge 19 commits into from

Conversation

@NicoHood
Copy link
Contributor

NicoHood commented Jan 20, 2016

Originally based on #3549

See the link above for additional recompile instructions of the different components and also the work of @ShorTie8 https://github.com/ShorTie8/Arduino_IDE

Take me to Download!

Just see the latest readme here which should point to the current downloads:
https://github.com/NicoHood/Arduino

Attention

This PR does include precompiled binaries! You should remove those before you merge this PR. I could do this on a different branch, if you are willing to merge this PR. But for now we keep this PR as visual reference of the changes I made to far to build.xml.

Adding precompiled binaries can be avoided if the arduino company uploads those files on their folders and also recompile new arm builds, such as the arduino-builder arm build (which needs a fix for now).

Build instructions

This PR can be build with an arm device or any x64 linux:

#x64
ant clean build dist -Dplatform=arm

#arm
ant clean build

To create new checksums use this:

shasum ./arduino-builder-arm-1.3.6.tar.bz2 | awk '{ print $1 }' > ./arduino-builder-arm-1.3.6.tar.bz2.sha

Compiling on arm with the binaries in place takes about 5~10min on a raspi2 which is acceptable since you do this once a day as dev.

Please note that crosscompiling the IDE is possible, since the binaries are precompiled on arm. The arm binaries are not yet all crosscompile able. Also it is not worth to add this option, since we only need to do that once (except for arduino-builder).

Bugs

Edit: Fixed! See below

The current build will not start! This happens with crosscompile and normal compilation.

arduino-1.6.8 $ ./arduino
Picked up JAVA_TOOL_OPTIONS: 
Tool name : avrdude 6.0.1-arduino5
     sys : i386-apple-darwin11 
     sys : x86_64-linux-gnu 
     sys : i686-linux-gnu 
     sys : i686-mingw32 
 seems to have no downloadable contributions for your operating system, but it is installed in
/home/pi/hackallthethings/arduino-1.6.8/hardware/tools/avr
Tool name : avr-gcc 4.8.1-arduino5
     sys : i386-apple-darwin11 
     sys : x86_64-linux-gnu 
     sys : i686-linux-gnu 
     sys : i686-mingw32 
 seems to have no downloadable contributions for your operating system, but it is installed in
/home/pi/hackallthethings/arduino-1.6.8/hardware/tools/avr
java.lang.NullPointerException
    at processing.app.BaseNoGui.createToolPreferences(BaseNoGui.java:851)
    at processing.app.BaseNoGui.initPackages(BaseNoGui.java:609)
    at processing.app.Base.<init>(Base.java:295)
    at processing.app.Base.guardedMain(Base.java:222)
    at processing.app.Base.main(Base.java:136)

This bug happens here:
https://github.com/NicoHood/Arduino/blob/master/arduino-core/src/cc/arduino/contributions/packages/ContributionsIndexer.java#L273

And might or might not be related to this:
6398cde

Which looks to me like it is a bug from the commit above:

public DownloadableContribution getDownloadableContribution(Platform platform) {

This commit caused the error, I reverted the commit:
6398cde

cc @cmaglie @facchinm @CRImier @imihalkovic @TXCodeDancer @Testato

NicoHood referenced this pull request Jan 20, 2016
This is a necessary step to fully support ARM builds of the Arduino IDE

See #3549
@ShorTie8
Copy link

ShorTie8 commented Jan 24, 2016

Ya, you broke it when you updated to 1.6.8, lol.
Maybe try syncing your git ??

@NicoHood
Copy link
Contributor Author

NicoHood commented Jan 24, 2016

It has to do something with this commit:
6398cde

I think we can easily fix this one. I just dont know how, but @cmaglie knows how this works. Then I get update the git again and it should compile then without any additional steps.

@ShorTie8
Copy link

ShorTie8 commented Jan 26, 2016

Yup, If you undo that patch, all is well.
I tried adding some stuff into package_index.json, but then it fails the sig check.
Which is some sort of funky file, so I can't make a new sig file.
It seems like there new motto is, DEATH to open source !!
ffissore was/is a big advocate of open source, the way the IDE is getting locked down with all those stupid exceptions and stuff, might be a cause for his moving on, but that is just MHO !!!!

They even make a hidden directory to hide the package_index.json and sig file

 locate package_index.json
/home/pi/.arduino15/package_index.json
/home/pi/.arduino15/package_index.json.sig
/usr/local/src/Arduino_IDE/package_index.json.diff

ps: @NicoHood your github could still use updating, there are no 'git tags' after 1.6.5-r5 .. :/~

@Testato
Copy link

Testato commented Jan 26, 2016

+1
Il 26/gen/2016 08:52, "ShorTie8" notifications@github.com ha scritto:

Yup, If you undo that patch, all is well.
I tried adding some stuff into package_index.json, but then it fails the
sig check.
Which is some sort of funky file, so I can't make a new sig file.
It seems like there new motto is, DEATH to open source !!
ffissore was/is a big advocate of open source, the way the IDE is getting
locked down with all those stupid exceptions and stuff, might be a cause
for his moving on, but that is just MHO !!!!


Reply to this email directly or view it on GitHub
#4457 (comment).

@NicoHood
Copy link
Contributor Author

NicoHood commented Jan 26, 2016

IDE is now fixed and a download is provided. Please test and report if it works on your device and which device you used.

The current master is broken, due to the lack of a new arduino-builder arm version. But I tagged the working version, so you can always go back to this and recompile it yourself.
https://github.com/NicoHood/Arduino

@cmaglie
Copy link
Member

cmaglie commented Jan 26, 2016

The patch on the HostDownloadable class has been made to allow updates of the cores from the Boards Manager. As you have noticed the packages_index.json still misses the tools section with the arm flavour of the tools, you can add it by yourself and rename package_index.json to test_package_index.json for testing (adding the test_ prefix will skip the signature check).

Without this patch the next time you try to update the AVR core, or install a non-AVR core, through the Board Manager the IDE will download the tools for x86 and consequently fail: that's the reason why I added the patch, not to lock someone out.

ffissore was/is a big advocate of open source, the way the IDE is getting locked down with all those stupid exceptions and stuff, might be a cause for his moving on, but that is just MHO !!!!

@ShorTie8
I can understand your frustration but, as you can see from the git history, the signature check has been added by ffissore himself, a long time ago, and for a very good reason. I can assure you that this is not the cause of his moving and I would like to ask you (and anyone else) to not make FUD out of personal choices that someone may have made for a number of different reasons that are out of your/our knowledge.

Thanks.

@cmaglie
Copy link
Member

cmaglie commented Jan 26, 2016

Adding precompiled binaries can be avoided if the arduino company uploads those files on their folders and also recompile new arm builds, such as the arduino-builder arm build (which needs a fix for now).

@NicoHood
We are going to setup an ARM builder on our CI system, we'll provide an ARM build shortly (at least for the arduino-builder it should be easy).

@NicoHood
Copy link
Contributor Author

NicoHood commented Jan 26, 2016

Oh I see now. Thx for clarifying @cmaglie

So what about arduino hosting the binaries? Then I can add the download link instead and remove the binaries from the git. This would make things simpler.

If the arm arduino-builder (with the path fix) is setup and the other files are hosted it should be easy to maintain this. It would be almost ready to merge. We just have to keep track of the changes and apply those additions for arm too. But most of the work is done already.

@cmaglie
Copy link
Member

cmaglie commented Jan 26, 2016

So what about arduino hosting the binaries?

Sure, but we can't host untrusted binaries, we need to reproduce the build here. Let's start with arduino-builder, I'll give it a spin tomorrow, now I should go AFK for a while.

@NicoHood
Copy link
Contributor Author

NicoHood commented Jan 26, 2016

Sure, thats true.
You can use the steps I used above (except for arduino builder, you know this one better). Then the checksum should be the same. Keep in mind that most arm stuff has to be compile on an arm device. RaspberryPi2 works perfect for me and in reasonable time (toolchain takes long and has a missing download mirror atm for all systems).

@ShorTie8
Copy link

ShorTie8 commented Jan 27, 2016

fyi: My script always builds the latest git versions, but may tar them up as a different version so the build.xml is satisfied. And I don't hide what it is doing, I'm nice and it tells ya in the push to be.

Thankz @NicoHood for updating and fixing github

ps @cmaglie it's my computer not yours, why do write files that I can not normally see ??
Like your trying to hide something that peeps can't see it !!!!!!!!
Thought this was open source and not find me if you can !!!!!
And if you look at what ffissore said a couple of months ago, it sure seems to me that some thing turn his stomach quickly to make him move on, but that is just MF`nHO.

So how do you make the sig file ???

@facchinm
Copy link
Member

facchinm commented Jan 27, 2016

@ShorTie8, to make the sig file you only need echo shasum $filename > $filename.sig.
But, please, stop acting like we hid something and stop talking about people you don't know. Everything in the build system is crystal clear, the utility to generate the signature is open source and the code to check if the downloaded file is valid is open source too.
If you are worried about your security you only need to read the code 😉

@ShorTie8
Copy link

ShorTie8 commented Jan 27, 2016

@facchinm oh really ??

pi Arduino_IDE # cat /home/pi/.arduino15/package_index.json.sig
▒V▒Y▒
        $▒k)B▒Yc▒H2▒▒8▒bEv▒=▒s▒▒W▒▒m▒zA▒y▒J*▒K▒▒▒:▒Ou,kT▒▒(LP▒▒▒K00i ▒▒Tn[▒?▒▒)▒[u▒}▒▒▒:]▒▒▒▒▒G▒k▒K▒t▒▒
▒▒      c▒#▒ǯI
              ▒Dhv%▒e▒▒*ߵu▒2 ▒▒/▒▒H▒a▒▒▒▒
㰑0P▒▒3▒▒▒?▒
            ▒P2▒}+▒▒▒▒.j▒▒Y\▒P▒▒▒)▒ů▒O▒▒:&▒▒z▒▒▒i▒▒▒▒o▒\▒▒l▒▒
@▒+6▒▒M▒▒
         ▒0▒▒j▒▒I▒▒▒▒▒▒w▒▒▒6▒(▒▒V▒el▒▒[▒▒Rn▒▒▒▒▒tn'▒▒▒(z▒▒K0▒▒▒▒▒▒XՃB▒Z▒M▒Ta▒▒P˿m▒)▒▒▒˥jr▒PMj▒▒T▒_}▒▒▒▒c(▒d▒J▒▒6▒▒g4▒(▒7ܢ▒▒▒▒N▒.6▒t٘B▒▒▒2▒▒▒▒▒-QfV▒R▒▒}d▒▒▒X▒M▒▒▒;▒▒▒▒▒H▒4q▒C▒▒Z▒▒▒▒▒w   &▒▒$Ǐq▒Q▒▒▒▒▒▒s▒@▒k▒r ▒▒▒▒t▒
▒/▒▒֎▒▒h▒▒~ڀX▒\▒r▒PuTTYpi Arduino_IDE # PuTTY
@ShorTie8
Copy link

ShorTie8 commented Jan 27, 2016

@cmaglie Actually it seems I think Board Manager is smart enough not to offer any platform specific downloads for unknown os types. Or so it seems because I was trying to add the sam stuff for yucks and couldn't find it.

But truely not sure, and not worth my time to figure further, I'll just hack the build.xml to include it if peeps want it and see how it goes .. :/~

So I got the IDE and toolchain-avr of course, arduino-builder, astyle, bossac, Coan, ctags, OpenOCD and listSerialPortsC all being compiled and/or options to. Some of witch I don't know what they are specifically for or where they might go .. :/~

What other os specific things need compiling ??

@cmaglie
Copy link
Member

cmaglie commented Jan 27, 2016

@ShorTie8 do you want to express your disappontment? that's fine, I'm just asking to not talk on behalf of someone else, especially if that someone else is unaware of that and what are you saying is just your opinion.

to make the sig file you only need echo shasum $filename > $filename.sig`

shasum is needed to make the .sha files needed on build.xml.

To make the .sig file we use gpg and a private key to sign the package_index.json file. This is a common way to use strong crypto to validate downloaded package and protect against tampering.
Just for the note:

  • the same approach is adopted by the major linux distributions (and the reason is clear, literature is full of cases when a single file compromised in an FTP server lead to thousand of user's PC infected).
  • we do the check above on our package index, but anyone is free to use his own package_xxxxx_index.json, like a lot of 3rd party producer, unaffiliated with arduino, does (see https://github.com/arduino/Arduino/wiki/Unofficial-list-of-3rd-party-boards-support-urls)

Hope this clarifies the reasons behind the .sig file.

@cmaglie
Copy link
Member

cmaglie commented Jan 27, 2016

@cmaglie Actually it seems I think Board Manager is smart enough not to offer any platform specific downloads for unknown os types. Or so it seems because I was trying to add the sam stuff for yucks and couldn't find it.

Yes that's the reason why I added the extra case for ARM, once the builds are in place everything should go smooth ;-)

@PaulStoffregen
Copy link
Contributor

PaulStoffregen commented Jan 29, 2016

I'm making good progress on the 2014q1 toolchain build. Details here: arduino/arduino-builder#105 (comment)

@NicoHood
Copy link
Contributor Author

NicoHood commented Jan 30, 2016

Thx for uploading the binaries. However the checksums did not match the checksums that I compiled myself. Maybe you had a different compiler version.

However I've updated the checksums, reverted the patch (for the json file) and updated the latest git. I also remove the binaries now.

@cmaglie The package_index.json does not seem to work for me. I still get this error, that no proper versions were found:

   83  cd arduino-1.6.8/
   85  cd dist/
   86  wget http://downloads.arduino.cc/packages/test_package_arm_index.json
   89  mv test_package_arm_index.json test_package_index.json 
   90  rm package_index.json

It seems to totally ignore the file (even if I delete it), so I do not know where it is searching. Or do I have to add this before compiling the IDE? Also could you please add the .sig file too (without the arm in the name, if that matters)?

Edit: I had to remove the file from .arduino15/package_index.json. However the test prefix does not skip the signing.

It also looks like this file totally differs from the current git: http://downloads.arduino.cc/packages/test_package_arm_index.json Also I could not apply the patch.

Then we need to merge the commit together (to remove the binaries from the history), test the IDE (on different systems) and also remove the additional readme that I added.

cmaglie added a commit to cmaglie/Arduino that referenced this pull request Feb 1, 2016
@cmaglie cmaglie self-assigned this Feb 1, 2016
@cmaglie
Copy link
Member

cmaglie commented Feb 1, 2016

@NicoHood
I've rebased and squashed your commits in this branch:
https://github.com/arduino/Arduino/compare/master...cmaglie:arm-2?expand=1

This lead to a very clean history as you described, and now I'm able to build the IDE on the raspberry and run it successfully.

The package_index.json does not seem to work for me.

Yes, sorry, I forget to tell you that you can set the java property PACKAGE_INDEX_URL to override the default URL used by the IDE to download the package index, BTW you're right this won't bypass the signature check so I added a this possibility here 0e471bc

After that you can overrides the package_index.json in this way: c35250d (note that this commit will be removed before merging the branch upstream).

Now I'm looking into ArduinoBot to allow it build also the ARM version. We are very close!

@NicoHood
Copy link
Contributor Author

NicoHood commented Feb 2, 2016

Great work done!
So I guess after ArduinoBot can also build ARM builds we are done and also dont need to maintain 3rd party arm builds? This would be totally fantastic!

I am going to test this later. We then could add a list of tested boards and check if all of them work properly. Also the diff itself is quite simple. It was a lot of work to get there, but actually its impressive that its not much to change. ;)

@PaulStoffregen
Copy link
Contributor

PaulStoffregen commented Feb 2, 2016

About 10 hours are left until the (hopefully) final build of the toolchain for Due & Zero is complete.
arduino/arduino-builder#105

@PaulStoffregen
Copy link
Contributor

PaulStoffregen commented Feb 3, 2016

Finished building and checking the 2014q1 toolchain. :)
arduino/arduino-builder#105 (comment)

@facchinm facchinm mentioned this pull request Feb 8, 2016
cmaglie added a commit to cmaglie/Arduino that referenced this pull request Feb 9, 2016
cmaglie added a commit to cmaglie/Arduino that referenced this pull request Feb 24, 2016
@theanch92
Copy link

theanch92 commented Mar 20, 2016

Hi, I am trying to install the Arduino Two board on Raspberry but at compile time I getting the message "arm-none-eabi-g ++ exec format error" even with the 2014q1 toolchain by PaulStoffregen. Can someone help me?

@PaulStoffregen
Copy link
Contributor

PaulStoffregen commented Mar 20, 2016

Can someone help me?

Maybe, but what steps would I need to take to recreate this error? I don't even know which RPi you're using. I can't see your screen, or what things you've click or what commands you've typed.

I do know that toolchain runs fine on my Raspberry Pi. I'm absolutely confident if I try it again the way I know how, it'll work perfectly, as it did a few weeks ago. Can you understand my reluctance to spend time on this without clear info about exactly the steps you took?

@theanch92
Copy link

theanch92 commented Mar 20, 2016

Thanks for the reply. If you say it works then definitely something I may be wrong. Meanwhile, I have a Raspberry pi 3, and I'm using the build of NicoHood 1.6.8-rc. Everything ok for Arduino Uno / Mega. I have the need to program for an Arduino Due, in the Arduino IDE. I go to Tools-> Board Manager and add "Sam Arduino Board 1.6.7". Once completed I select the Arduino card Two and I try to compile I get error:
fork / exec /home/pi/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++: exec format error.
Now I try to replace "arm-none-eabi-gcc" in /home/pi/.arduino15/packages/arduino/tools/arm-none-eabi-gcc with your build for ARM.
But I have always the same error:
fork / exec /home/pi/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++: exec format error.

Definitely wrong in something .. and I apologize for my stupidity

@PaulStoffregen
Copy link
Contributor

PaulStoffregen commented Mar 20, 2016

On your RPi in a terminal, can you try these command, and post the result?

md5sum /home/pi/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++
file /home/pi/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++

When I run them here on my Raspberry Pi, with the pathname to that file, I see this:

paul@raspberrypi ~ $ md5sum gcc-arm-none-eabi-4_8-2014q1/bin/arm-none-eabi-g++
bfd196cbc3e85ec922d8d5d7fa8d5449  gcc-arm-none-eabi-4_8-2014q1/bin/arm-none-eabi-g++
paul@raspberrypi ~ $ file gcc-arm-none-eabi-4_8-2014q1/bin/arm-none-eabi-g++
gcc-arm-none-eabi-4_8-2014q1/bin/arm-none-eabi-g++: ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.26, BuildID[sha1]=0x539864a83fc520418c0686b9d66e9da1a2f5fd44, stripped
@theanch92
Copy link

theanch92 commented Mar 20, 2016

I do not know what strange reason, but does not copy the files .. they do not matched the md5! Now it works beautifully! Thank you so much for your patience, I owe you a drink

cmaglie added a commit to cmaglie/Arduino that referenced this pull request Apr 8, 2016
@cmaglie
Copy link
Member

cmaglie commented Apr 8, 2016

Supersed by #4517

@cmaglie cmaglie closed this Apr 8, 2016
@cmaglie cmaglie added this to the Release 1.6.9 milestone Apr 8, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

8 participants
You can’t perform that action at this time.