Incorrect "Sketch too big" error on insufficient Flash space when uploading using programmer

[deltaphi]

Given a sketch of compiled size X, where X by itself will fit into the flash of a given chip, but X plus the Arduino Bootloader will not. When using the regular "Upload" Button, the Arduino IDE correctly prints a warning about insufficient flash memory and aborts the upload process.

However, when using "Upload using Programmer" (precisely to use the extra space by getting rid of the bootloader), the Arduino IDE again prints this warning (this time incorrectly!) and aborts the upload process.

To resolve this issue, the IDE should take the upload method (Bootloader/Programmer) into account when computing the amount of flash available on the chip.

I verified this using Arduino 1.0.4 and Arduino 1.0.5, both times compiling for an ATmega8 ("Ardunio NG or older w/ATmega8").

Until this bug is resolved, I would also appreciate a hint on how to work around this issue (i.e., how to burn an "oversize" sketch with a programmer) in the meantime.

[cmaglie]

Thanks for the bug report.
As a temporary workaround you can increase the memory limit by changing the boards.txt file into hardware/arduino/boards.txt of your arduino installation, and restarting the IDE afterwards.

https://github.com/arduino/Arduino/blob/master/hardware/arduino/boards.txt#L511

C

[per1234]

It seems like it would be a bad idea to allow random application code to be flashed into the boot section. Even when you are doing an "Upload Using Programmer", the BOOTRST fuse is still programmed. Am I wrong?

[matthijskooijman]

It seems like it would be a bad idea to allow random application code to be flashed into the boot section. Even when you are doing an "Upload Using Programmer", the BOOTRST fuse is still programmed. Am I wrong?

I think you are correct here. If we'd want to allow using the full flash space by using a programmer, the fuses must also be changed, which would be a significant change (though writing the fuses using "Upload using programmer" would not neccesarily be bad, since currently people regularly get surprised by the fact they their CPU speed is wrong when they did "Upload using programmer" on a new chip directly, without doing "Burn bootloader" once).

[deltaphi]

It seems like it would be a bad idea to allow random application code to be flashed into the boot section.

In my usecase, there actually is no boot section. If I remember the project correctly, I had simply run out of space and needed the extra bytes for the bootloader. As I was operating on a custom board with an Arduino inside (i.e., not destined for constant reprogramming) using an external programmer was totally fine.

Also, similarly remembering the project, "Upload using bootloader" actually does remove the bootloader, so I really did get the extra space.

[matthijskooijman]

In my usecase, there actually is no boot section.

If you have a custom board that is never intended to be used with a bootloader, the boards.txt for that board can both set the fuses (at least on AVR) to disable the bootloader and the max flash size can be set to the full flash size, and this problem will not exist (but uploading using the bootloader will not be available, unless there is some ROM bootloader that takes no space).

But if the boards.txt lists the fuses to enable the bootloader section, then no application code can be put into the bootloader section (not even when uploading using the programmer, since the fuses cause the chip to reset to the bootloader area rather than the sketch, so if there was application code in the bootloader area, that would start executing somewhere near the end of the sketch. The only, albeit hacky, way to make this work without changing the fuses is to leave the bootloader area empty, so the chip just skips over all 0xff bytes there at startup and then wraps around to the sketch reset vector at 0x0).

Most of this is AVR-specific and might or might not apply to other architectures.

In general, the original request seems to be (or could be) to allow different configurations (fuses and max flash) for when uploading by bootloader or by programmer. But for AVR, for this to really work, the fuses must also be programmed when doing "upload using programmer".

[per1234]

avrdude at least supports programming fuses and flash in a single command:

tools.avrdude.program.pattern="{cmd.path}" "-C{config.path}" {program.verbose} {program.verify} -p{build.mcu} -c{protocol} {program.extra_params} -Uhfuse:w:{program.high_fuses}:m  "-Uflash:w:{build.path}/{build.project_name}.hex:i"

so the only thing that would be needed is an property equivalent to upload.maximum_size for use with the program action.

The current workaround for this in the 3rd party platform world is to add a "no bootloader" custom board option (example)
but that's not quite so user friendly as the program action handling everything automatically.

[deltaphi]

In my usecase, there actually is no boot section.

If you have a custom board that is never intended to be used with a bootloader, the boards.txt for that board can both set the fuses (at least on AVR) to disable the bootloader and the max flash size can be set to the full flash size, and this problem will not exist (but uploading using the bootloader will not be available, unless there is some ROM bootloader that takes no space).

Thinking back to the question that raised this bugreport: I had a custom circuit soldered onto a perfboard, including the typical Arduino Uno parts (Xtal, Reset button, ISP Header, USB-Serial Adapter Header). I socketed store-bought Atmega328s into those boards. For the project I actually preferred not having the bootloader, since the 1 second delay at power on would have been annoying. Needing the flash space for the project was a problem on top, which eventually prompted this bugreport. The entire project was originally inspired by the "Arduino on Breadboard" and "DIY Arduino on Perfboard" post surrounding the Arduino community at the time.

As I had store-bought AVRs, I must have programmed the fuses somehow, as the factory defaults don't work for an Arduino Uno board. I remember running into the issue of "Upload using programmer" not setting the fuses and incorrectly writing up a whole bunch of the AVRs as defective. Although whether I set the fuses using avrdude directly, using a custom boards.txt or messing with an existing boards.txt, I cannot say anymore.

In any case, I was still working on on the code when I had set up circuit on the perfboard (removing bugs, adapting pinlayouts to circuit board revisions, adding the occasional functionality and the like). So this was not a matter of "mess with avrdude once" but of repeatedly recompiling my code and uploading through the programmer.