Provide file format I: Intel HEX with comments that ignores checksum errors #1030

stefanrueger · 2022-07-16T22:48:18Z

The new file type I is essentially Intel HEX that, on download, inserts comments next to data records with the resolved effective address and an ASCII dump of that same record. On upload the I format is permissive with respect to check sum errors, which is useful, eg, after manipulating an Intel HEX file for debugging.

$ avrdude -c usbtiny -qqp m328p -U flash:w:blink.hex:I
$ avrdude -c usbtiny -qqp m328p -U flash:r:-:I | tee b.hex
:2000000033C0000041C000003FC000003DC000003BC0000039C0000037C0000035C0000010 // 00000> 3@..A@..?@..=@..;@..9@..7@..5@..
:2000200033C0000031C000002FC000002DC000002BC0000043C0000027C0000025C0000046 // 00020> 3@..1@../@..-@..+@..C@..'@..%@..
:2000400023C0000021C000001FC000001DC000001BC0000019C0000017C0000015C00000C0 // 00040> #@..!@...@...@...@...@...@...@..
:2000600013C0000011C0000011241FBECFEFD8E0DEBFCDBF21E0A0E0B1E001C01D92A230D7 // 00060> .@...@...$.>OoX`^?M?!`_`1`.@.."0
:20008000B207E1F755D068C0BBCF1092800082E080938100259AE0E6F1E0E49147D06C1979 // 00080> 2.awUPh@;O.....`....%.`fq`d.GPl.
:2000A0007D098E099F09653F714081059105B0F33DD06B017C011D9AF1CF08950F930FB7F5 // 000a0> }.....e?q@....0s=Pk.|...qO.....7
:2000C0000F930AB50F5F0ABD70F00BB50F4F0BBD50F0009100010F4F0093000100910101ED // 000c0> ...5._.=pp.5.O.=Pp.....O........
:2000E0000F4F009301010F910FBF0F9118952FB7F894609184006091850036B37AB58BB59D // 000e0> .O.......?..../7x.`...`...63z5.5
:20010000909100012FBF2091010130FF06C06F3F21F07F5F8F4F9F4F2F4F33E02695979546 // 00100> ..../?_...0..@o?!p._.O.O/O3`&...
:200120008795779567953A95C9F70895E0CF0895789483E084BD85BD81E080936F0094E0DA // 00120> ..w.g.:.Iw..`O..x..`.=.=.`..o..`
:200140009093B1008093B00087E880937A001092C1009BDFB2DFFECFF89400C0F894FFCF2B // 00140> ..1...0..h..z...A.._2_~Ox..@x..O
:1801600048656C6C6F2C20776F726C640A626C696E6B2E68657800FF93                 // 00160> Hello,_world.blink.hex..
:00000001FF
$ avrdude -c usbtiny -qqp m328p -U flash:w:b.hex:I

…errors The new file type I is essentially Intel HEX that, on download, inserts comments next to data records with the resolved effective address and an ASCII dump of that same record. On upload the `I` format is permissive with respect to check sum errors, eg, after manipulated an Intel HEX file for debugging.

mcuee · 2022-07-17T01:15:24Z

Looks like a useful enhancement.

mcuee · 2022-07-17T06:11:06Z

@stefanrueger
Somehow I do not see the comments in the following output under MSYS2 mingw64. Maybe I need to under better what you mean by inserts comments next to data records with the resolved effective address and an ASCII dump of that same record.

$ ./avrdude.exe -v

avrdude.exe: Version 7.0-20220716 (79921e5)
             Copyright (c) Brian Dean, http://www.bdmicro.com/
             Copyright (c) Joerg Wunsch

             System wide configuration file is "C:/work/avr/avrdude_test/avrdude_sr/
build_mingw64_nt-10.0-22000/src/avrdude.conf"


avrdude.exe: no programmer has been specified on the command line or the config file
             Specify a programmer using the -c option and try again


$ ./avrdude -c usbasp -qqp m328p -U flash:w:blinky_uno.hex:I

$ ./avrdude -c usbasp -qqp m328p -U flash:r:-:I | tee b_uno.hex
:200000000C945C000C946E000C946E000C946E000C946E000C946E000C946E000C946E0082 // 00000> ..\...n...n...n...n...n...n...n.
:200020000C946E000C946E000C946E000C946E000C946E000C946E000C946E000C946E0050 // 00020> ..n...n...n...n...n...n...n...n.
:200040000C9413010C946E000C946E000C946E000C946E000C946E000C946E000C946E008A // 00040> ......n...n...n...n...n...n...n.
:200060000C946E000C946E0000000000240027002A0000000000250028002B000404040467 // 00060> ..n...n.....$.'.*.....%.(.+.....
:200080000404040402020202020203030303030301020408102040800102040810200102F1 // 00080> ....................._@......_..
:2000A00004081020000000080002010000030407000000000000000011241FBECFEFD8E063 // 000a0> ..._.....................$.>OoX`
:2000C000DEBFCDBF21E0A0E0B1E001C01D92A930B207E1F70E945D010C94CC010C940000FE // 000c0> ^?M?!`_`1`.@..)02.aw..]...L.....
:2000E000E1EBF0E02491EDE9F0E09491E9E8F0E0E491EE23C9F0222339F0233001F1A8F4B5 // 000e0> akp`$.mip`..ihp`d.n#Ip"#9p#0.q(t
:20010000213019F1223029F1F0E0EE0FFF1FEE58FF4FA591B4912FB7F894EC91811126C0B7 // 00100> !0.q"0)qp`n...nX.O%.4./7x.l...&@
:2001200090959E239C932FBF08952730A9F02830C9F0243049F7209180002F7D03C0209139 // 00120> ...#../?..'0)p(0Ip$0Iw_.../}.@_.
:2001400080002F7720938000DFCF24B52F7724BDDBCF24B52F7DFBCF2091B0002F77209385 // 00140> ../w_..._O$5/w$=[O$5/}{O_.0./w_.
:20016000B000D2CF2091B0002F7DF9CF9E2BDACF3FB7F8948091050190910601A0910701ED // 00160> 0.RO_.0./}yO.+ZO?7x........._...
:20018000B091080126B5A89B05C02F3F19F00196A11DB11D3FBFBA2FA92F982F8827BC01A6 // 00180> 0...&5(..@/?.p..!.1.??:/)/./.'<.
:2001A000CD01620F711D811D911D42E0660F771F881F991F4A95D1F708958F929F92AF92C3 // 001a0> M.b.q.....B`f.w.....J.Qw....../.
:2001C000BF92CF92DF92EF92FF920E94B8004B015C0184EFC82EDD24D394E12CF12C0E944A // 001c0> ?.O._.o.....8.K.\..oH.]$S.a,q,..
:2001E000B800681979098A099B09683E734081059105A8F321E0C21AD108E108F10888EEEF // 001e0> 8.h.y.....h>s@....(s!`B.Q.a.q..n
:20020000880E83E0981EA11CB11CC114D104E104F10429F7FF90EF90DF90CF90BF90AF9097 // 00200> ...`..!.1.A.Q.a.q.)w..o._.O.?./.
:200220009F908F9008951F920F920FB60F9211242F933F938F939F93AF93BF938091010157 // 00220> ...........6...$/.?...../.?.....
:2002400090910201A0910301B09104013091000123E0230F2D3758F50196A11DB11D209381 // 00240> ...._...0...0...#`#.-7Xu..!.1._.
:2002600000018093010190930201A0930301B09304018091050190910601A0910701B0910A // 00260> .........._...0..........._...0.
:2002800008010196A11DB11D8093050190930601A0930701B0930801BF91AF919F918F9188 // 00280> ....!.1........._...0...?./.....
:2002A0003F912F910F900FBE0F901F90189526E8230F0296A11DB11DD2CF789484B582601B // 002a0> ?./....>......&h#...!.1.ROx..5.`
:2002C00084BD84B5816084BD85B5826085BD85B5816085BD80916E00816080936E00109244 // 002c0> .=.5.`.=.5.`.=.5.`.=..n..`..n...
:2002E000810080918100826080938100809181008160809381008091800081608093800069 // 002e0> .......`.........`.........`....
:200300008091B10084608093B1008091B00081608093B00080917A00846080937A00809101 // 00300> ..1..`..1...0..`..0...z..`..z...
:200320007A00826080937A0080917A00816080937A0080917A00806880937A001092C10078 // 00320> z..`..z...z..`..z...z..h..z...A.
:20034000EDE9F0E02491E9E8F0E08491882399F090E0880F991FFC01E859FF4FA591B49192 // 00340> mip`$.ihp`...#.p.`....|.hY.O%.4.
:20036000FC01EE58FF4F859194918FB7F894EC91E22BEC938FBFC0E0D0E081E00E947000C5 // 00360> |.nX.O.....7x.l.b+l..?@`P`.`..p.
:1C0380000E94DD0080E00E9470000E94DD002097A1F30E940000F1CFF894FFCFEA         // 00380> ..]..`..p...]._.!s....qOx..O
:00000001FF

mcuee · 2022-07-17T09:15:37Z

Similarly for Linux.

mcuee@UbuntuSwift3:~/build/avr/avrdude_bin$ ./avrdude_pr1030 -c usbasp -qqp m168p 
-U flash:w:./Blink.ino_atmega168p_16000000L.hex:I 
mcuee@UbuntuSwift3:~/build/avr/avrdude_bin$ ./avrdude_pr1030 -c usbasp -qqp m168p -U flash:r:-:I 
| tee blink168p.hex
:200000000C9461000C9473000C9473000C9473000C9473000C9473000C9473000C9473005A // 00000> ..a...s...s...s...s...s...s...s.
:200020000C9473000C9473000C9473000C9473000C9473000C9473000C9473000C94730028 // 00020> ..s...s...s...s...s...s...s...s.
:200040000C948D000C9473000C9473000C9473000C9473000C9473000C9473000C947300EE // 00040> ......s...s...s...s...s...s...s.
:200060000C9473000C9473000000000900030200000405080000000000000000000000013A // 00060> ..s...s.........................
:200080000204081020408001020408102001020408102040804004040404040404040202C0 // 00080> ...._@......_....._@.@..........
:2000A0000202020203030303030302020300000000250028002B0000000000240027002A32 // 000a0> .................%.(.+.....$.'.*
:2000C000000011241FBECFEFD4E0DEBFCDBF21E0A0E0B1E001C01D92A930B207E1F70E94E5 // 000c0> ...$.>OoT`^?M?!`_`1`.@..)02.aw..
:2000E000F1010C9401020C94000061E08DE00C94860161E08DE00E94C20168EE73E080E0DA // 000e0> q.........a`.`....a`.`..B.hns`.`
:2001000090E00E94FC0060E08DE00E94C20168EE73E080E090E00C94FC001F920F920FB693 // 00100> .`..|.``.`..B.hns`.`.`..|......6
:200120000F9211242F933F938F939F93AF93BF938091050190910601A0910701B0910801AB // 00120> ...$/.?...../.?........._...0...
:200140003091040123E0230F2D3768F126E8230F0296A11DB11D209304018093050190938F // 00140> 0...#`#.-7hq&h#...!.1._.........
:200160000601A0930701B09308018091000190910101A0910201B09103010196A11DB11D20 // 00160> .._...0..........._...0.....!.1.
:200180008093000190930101A0930201B0930301BF91AF919F918F913F912F910F900FBECD // 00180> ........_...0...?./.....?./....>
:2001A0000F901F9018950196A11DB11DD4CF3FB7F8948091000190910101A0910201B09152 // 001a0> ........!.1.TO?7x........._...0.
:2001C000030126B5A89B05C02F3F19F00196A11DB11D3FBFBA2FA92F982F8827BC01CD01DE // 001c0> ..&5(..@/?.p..!.1.??:/)/./.'<.M.
:2001E000620F711D811D911D42E0660F771F881F991F4A95D1F708958F929F92AF92BF9200 // 001e0> b.q.....B`f.w.....J.Qw....../.?.
:20020000CF92DF92EF92FF926B017C010E94D7004B015C01C114D104E104F104E9F00E94F0 // 00200> O._.o...k.|...W.K.\.A.Q.a.q.ip..
:2002200000020E94D700681979098A099B09683E73408105910570F321E0C21AD108E10892 // 00220> ....W.h.y.....h>s@....ps!`B.Q.a.
:20024000F10888EE880E83E0981EA11CB11CC114D104E104F10429F7FF90EF90DF90CF9076 // 00240> q..n...`..!.1.A.Q.a.q.)w..o._.O.
:20026000BF90AF909F908F900895789484B5836084BD85B5836085BDEEE6F0E080818160B7 // 00260> ?./.......x..5.`.=.5.`.=nfp`...`
:200280008083E1E8F0E082E08083808181608083E0E8F0E0808181608083E1EBF0E080815E // 00280> ..ahp`.`.....`..`hp`...`..akp`..
:2002A00084608083E0EBF0E080818160808387E880937A001092C1000895843081F028F49A // 002a0> .`..`kp`...`...h..z...A....0.p(t
:2002C0008230E9F08330F9F00895883071F0893091F08530B9F4809180008F7D03C0809134 // 002c0> .0ip.0yp...0qp.0.p.09t.....}.@..
:2002E00080008F778093800008958091B0008F778093B00008958091B0008F7DF9CF84B553 // 002e0> ...w........0..w..0.....0..}yO.5
:200300008F7784BD089584B58F7DFBCFCF93DF9390E0FC01E158FF4F24918A569F4FFC01A2 // 00300> .w.=...5.}{OO._..`|.aX.O$..V.O|.
:2003200084918823C9F090E0880F991FFC01E954FF4FA591B491FC01E355FF4FC591D49144 // 00320> ...#Ip.`....|.iT.O%.4.|.cU.OE.T.
:2003400061110DC09FB7F8948C91209582238C938881282328839FBFDF91CF91089562308A // 00340> a..@.7x..._..#....(#(..?_.O...b0
:2003600051F49FB7F8943C91822F809583238C93E8812E2BEFCF8FB7F894EC912E2B2C9317 // 00360> Qt.7x.<../...#..h..+oO.7x.l..+,.
:200380008FBFEACF1F93CF93DF93282F30E0F901E859FF4F8491F901E158FF4FD491F901EB // 00380> .?jO..O._.(/0`y.hY.O..y.aX.OT.y.
:2003A000EA56FF4FC491CC23A9F0162F81110E945D01EC2FF0E0EE0FFF1FE355FF4FA59139 // 003a0> jV.OD.L#)p./....].l/p`n...cU.O%.
:2003C000B4918FB7F894EC91111108C0D095DE23DC938FBFDF91CF911F910895DE2BF8CF8F // 003c0> 4..7x.l....@P.^#\..?_.O.....^+xO
:2003E00008950E9435010E94F0010E947500C0E0D0E00E9479002097E1F30E940000F9CF7E // 003e0> ....5...p...u.@`P`..y._.as....yO
:060400000895F894FFCFFF                                                     // 00400> ..x..O
:00000001FF

MCUdude · 2022-07-17T17:54:38Z

Thanks for the PR @stefanrueger! A char dump along with the hex file, and "support" for incorrect checksums might be useful sometimes.

The :a option doesn't like invalid checksums though...

$ ./avrdude -cusbtiny -patmega2560 -Uflash:w:/var/folders/6l/ypg6qbw172v1s4vtt6g990tw0000gn/T/arduino_build_218855/eeprom_get.ino.with_bootloader.hex:a -B1

avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude: Device signature = 0x1e9801 (probably m2560)
avrdude: NOTE: "flash" memory has been specified, an erase cycle will be performed
         To disable this feature, specify the -D option.
avrdude: erasing chip
avrdude: reading input file "/var/folders/6l/ypg6qbw172v1s4vtt6g990tw0000gn/T/arduino_build_218855/eeprom_get.ino.with_bootloader.hex"
avrdude: input file /var/folders/6l/ypg6qbw172v1s4vtt6g990tw0000gn/T/arduino_build_218855/eeprom_get.ino.with_bootloader.hex auto detected as Intel Hex
avrdude: ERROR: checksum mismatch at line 2 of "/var/folders/6l/ypg6qbw172v1s4vtt6g990tw0000gn/T/arduino_build_218855/eeprom_get.ino.with_bootloader.hex"
avrdude: checksum=0xc2, computed checksum=0xc1
avrdude: read from file '/var/folders/6l/ypg6qbw172v1s4vtt6g990tw0000gn/T/arduino_build_218855/eeprom_get.ino.with_bootloader.hex' failed

avrdude done.  Thank you.

I would assume it would detect the intel hex file format, find the incorrect checksum and treat is as :I?

dl8dtl · 2022-07-17T19:09:11Z

I'm not so sure whether automatically deducing a checksum error (for :a) to "the user might probably have wanted it that way" is a Good Thing.
Overall, I think the added comment is great, but I'm tempted to require a further user interaction for accepting failed checksum files. Specifying the :I probably classifies as "user interaction", assuming that the normal :a / :i would accept an extended hex file with comments but correct checksum (i.e. when reading files, the :I is only needed to override the checksum test).

stefanrueger · 2022-07-18T11:17:27Z

@MCUdude Yes, the suggested patch does not investigate in :a mode whether it's a strict Intel HEX file or a relaxed one with the comments in. Because the only difference between the two formats on upload is that checksums are ignored, I thought it safer to subsume :I under :i for autodetection (which, crucially, does not ignore checksums). @dl8dtl is correct in his assumptions/reading of what the proposed PR does.

BTW, an alternative to introducing a new file type :I would be the introduction of file type options, say -U flash:r:-:i/h for show me a human-readable variant of Intel HEX or -U eeprom:w:mods.eep:i/c to ignore checksum errors in Intel HEX file uploads. I plan to propose / suboptions (aka developer or advanced options) in -p arguments (to output what AVRDUDE knows about a particular part), so :i/h might come handy.

Another /h add-on could be to insert spaces between byte count, offset address, record type, data and data checksum just like:

:18 0160 00 48656C6C6F2C20776F726C640A626C696E6B2E68657800FF 93   // 00160> Hello,_world.blink.hex..

The seemingly redundant address in the comment bit is still useful because it is the effective address whilst the lhs address is an offset address to a base address that might have been specified earlier. If we think these spaces are useful, I would also change the :i read-in routine to ignore spaces between byte count, offset address, record type, data and data checksum, so that any :i/h produced output can be read with :i without needing to specify the /h.

Use cases. (@mcuee Just to clarify that the comments inserted into the Intel HEX file are the added columns that look like // 00160> Hello,_world.blink.hex..) One use case is where the EEPROM contents needs changing using simple tools like editors. The idea is that a download of the EEPROM with :I generates comments that tell you the effective address and gives you some orientation by displaying strings in the ASCII dump. When a user changes the Intel HEX record and tries to upload the modified file with :I then the upload will still succeed despite potentially wrong checksums. Another use case for :I is that you might want to see what firmware is on the board. With -U flash:r:-:I you get a sporting chance of spotting strings, application names or version strings.

Fun facts. Whilst wikipedia seems to think that '//' comments in Intel HEX files are often used in extensions of the format the format guru srec rejects the :I produced files with comments. Wikipedia also seems to think any extension of Intel HEX should not use the colon anywhere, presumably because some Intel HEX parsers ignore all characters after a record (in particular newline) and scan forward to the next colon. These two observations led me to tilt in favour of a new file type :I that is like Intel HEX rather than touching what :i does. Note :I can produce a colon in the ASCII dump, but when read by AVRDUDE with :i this does not matter as AVRDUDE ignores anything in the line after the checksum right up to line end (or file end if the last line does not have a line end char). It also does not matter to srec as it rejects Intex HEX files with comments regardless.

stefanrueger · 2022-07-23T17:57:26Z

What's the preference? :I (as is) or :i/h and :i/c separately?

MCUdude · 2022-07-25T21:39:09Z

What's the preference? :I (as is) or :i/h and :i/c separately?

I'd prefer :I. It's less typing and easier to memorize.

mcuee · 2022-07-26T09:25:49Z

What's the preference? :I (as is) or :i/h and :i/c separately?

I'd prefer :I. It's less typing and easier to memorize.

Same here.

stefanrueger · 2022-07-27T00:00:36Z

OK, so no changes needed here. This PR is then good to go. Thanks for looking at it, @dl8dtl, @MCUdude and @mcuee.

stefanrueger added the enhancement New feature or request label Jul 16, 2022

stefanrueger merged commit 7730706 into avrdudes:main Aug 2, 2022

stefanrueger deleted the ihexcomments branch August 2, 2022 17:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provide file format I: Intel HEX with comments that ignores checksum errors #1030

Provide file format I: Intel HEX with comments that ignores checksum errors #1030

stefanrueger commented Jul 16, 2022

mcuee commented Jul 17, 2022

mcuee commented Jul 17, 2022 •

edited

mcuee commented Jul 17, 2022

MCUdude commented Jul 17, 2022

dl8dtl commented Jul 17, 2022

stefanrueger commented Jul 18, 2022

stefanrueger commented Jul 23, 2022

MCUdude commented Jul 25, 2022

mcuee commented Jul 26, 2022

stefanrueger commented Jul 27, 2022

Provide file format I: Intel HEX with comments that ignores checksum errors #1030

Provide file format I: Intel HEX with comments that ignores checksum errors #1030

Conversation

stefanrueger commented Jul 16, 2022

mcuee commented Jul 17, 2022

mcuee commented Jul 17, 2022 • edited

mcuee commented Jul 17, 2022

MCUdude commented Jul 17, 2022

dl8dtl commented Jul 17, 2022

stefanrueger commented Jul 18, 2022

stefanrueger commented Jul 23, 2022

MCUdude commented Jul 25, 2022

mcuee commented Jul 26, 2022

stefanrueger commented Jul 27, 2022

mcuee commented Jul 17, 2022 •

edited