Q: Is this still an active project to be used with HP Prime G2?

[gecko65]

Sorry more a question then a error report. Is that still an active project I? I would try to build it on a current Linux dist.

[debrouxl]

You could indeed try to build it on a current Linux distro, the code isn't supposed to contain weird constructs which ought to break with newer toolchains.
However, I haven't worked on it or used it for years. I don't know whether the Prime G2 still supports the old protocol implemented by hplp.

[gecko65]

So give me some time to check that. I will come back here and report, once I got that working.
Thanks for your answer. Have a nice Advent season.

[BeatSkip]

@gecko65 any updates? Is it still working? I'm till looking for something to base my own library on

[debrouxl]

Your own use case might be different, e.g. embedded software / platform, but FWIW, nowadays, in order to maximize the usability for most end users, the best way to build software for [high-powered computer in the general sense] <- USB -> [graphing calculator, and many other USB devices] communication is to produce something based on WebUSB. The TI graphing calculators community has started doing that.

Beyond a bit of glue code, WebUSB-based software needn't be written in JS / TS: various other languages can be targeted to WebAssembly :)

[BeatSkip]

Well, so funnily enough, that's what I'm trying to achieve (an actually usable ppl and python IDE in c# Blazor webassembly with webusb).but to do that I need a reliable baseline communication library that I can copy the protocol/details off of as reverse engineering the entire workings is just too enthusiastic of a goal haha

Atm I have basic communication working (I can send a message to my prime) . But a working library will make life a lot easier. As even simple stuff like "how do I get a list of applications on the prime?" seems not too widely documented and any documentation spans over 8 years without any details on current status in modern firmware. (is it me or has the community on the prime died a bit? Such potential in this device, especially the G2)

[gecko65]

@BeatSkip Unfortunately so far I did not get it working. I could built it. But once I try to connect the calculator with my linux laptop teh program does not find any valid connection.

[gecko65]

I could see the hp prime G2 in the list of usb devices but hplp seams not to catch it. In the past I have had a sync problem with Palm and Linux. The solution was that first a initial transfer was needed with the Windows program and then the linux sync was able to communicate with the device. I know this is completely something different but maybe the hp prime also needs such an initialization for the communication?

[BeatSkip]

@gecko65 As i indicated before, i'm actually looking to bypass platforms entirely and go directly to Webassembly/Js with WebHID. Initial tests seem promising, as soon as i've got a basic proof of concept working i'll put up a public repo and leave a link here. you can keep an eye out for that if you're interested. currently i'm just using this code as a rough baseline indication and use PrimeComm as the actual basis to work off of as that seems to mostly be working with the G2 ( at least my tests for now seem to work). It's just that functionality in PrimeComm is very limited.

[debrouxl]

See also #5 (new USB PID) and #4 (newer variant of the protocol).

[gecko65]

@BeatSkip Yes I am in deed interested in the results of your future repo.

[gecko65]

@debrouxl Should I start an issue with the error/logging message of test_hpcalcs?

[metawops]

I, too, experimented with this in 2020 because my plan was to directly connect a Prime via cable to an Apple iPad Pro's USB-C port. I wanted to make an app that was able to take screenshots from the Prime's screen into the iCloud Photo Library. – Never was successful with that. However, Cyrille de Brebisson – who was an HP Prime team member – was very open about technical details regarding the USB protocol and supplied me with details in emails. This was in March 2020. I'm quite sure he would be supportive today, too. Just wanted to drop this in, in case somebody needs help.
Besides from that I'd very much be interested in testing any efforts made here. My platform is macOS & most of the time Safari (which doesn't yet support WebUSB). But I have Chrome & Firefox, too. => I'll watch this space or any folloing/spawned repositories. 😉

[debrouxl]

@gecko65 : for the benefit of others, indeed, but first of all, try modifying the code ( libhpcalcs/src/hplibs.h & libhpcalcs/src/link_prime_hid.c ) to look for the USB PID indicated in the lsusb output on your computer, in addition to (or instead of, but that's less good) the existing ones. In #5, @metawops mentioned a 0x2441 USB PID.

[adriweb]

@metawops: would you be able to share the/some information from Cyrille? I don't know how easy it would be to reach him nowadays considering HP Prime development got transferred out of HP. Although maybe Lionel got some from him already, but I'm not so sure. And that would have been much older than 2020.

[BeatSkip]

@metawops that would definitely be great! I'm almost done setting up my base repo. Hope to get it live tonight with a working simple communication message from the browser and I'm planning on taking it from there. But an open source up-to-date communication platform is definitely something the prime needs to get it's community a bit more lively with more people. And input from the devs would definitely be appreciated

[gecko65]

@debrouxl Yes I also have a 0x2441 USB PID. I changed that in the code (thanks for you deatilled hint). The behaviour already changed but I still do not get something like a menu. I am still checking if I missed something.

[gecko65]

@debrouxl Ok got it working - at least now as root (1st step) I get the menu. Now testing the test functionality :)

[debrouxl]

Good to hear that you can at least get the menu :)

[BeatSkip]

for those interested, i've got the repo up
https://github.com/BeatSkip/PrimeDev
now working to integrate my local test to work with the WebHID library instead of the Windows HidLibrary. But anyone interested should feel free to engange and pitch in :)

[metawops]

Let me try to reach Cyrille first (already sent out a ping) and check if it's okay to publish his infos here. When I asked him for those infos I promised I wouldn't publish it anywhere ... But I'll check old eMails, too, and see if we talked about this, too. Maybe he mentioned that it would be fine to publish it and I forgot about that. I'll report back with news.

[gecko65]

@debrouxl now sending a chat (Hello World) works. Receiving a chat works in the DUMP. I get a
00 F2 01 00 00 00 <LEN> <CHAR1LOW> <CHAR1HIGH> ....

[BeatSkip]

@gecko65 FYI, I managed to get the initial message to my prime working! What's your status? Any updates? And @metawops did you hear back from cyrille? Any more info on the protocol would now be appreciated as I've got the initial communication working in the browser.

[gecko65]

@BeatSkip : Yes as I already wrote I was able to send "chat" messages to the Prime. Receiving chat messages work only as a DUMP (need to check that).

[metawops]

@metawops did you hear back from cyrille?

Not yet, unfortunately, no.

[BeatSkip]

@gecko65 getting the messages back is pretty simple in principle.
Not sure how this couples into hplp but in my tool I've got it working byparsing the return data as:
Data[0] = 0x00
Data[1] = 0xF2 (data packet identifier, 0xF2 is 'chat message')
Data[2] = 0x03 (docs say count of something, but primecomm and hplp are looking for this as fixed 0x01, but with modern g2 appears to be always 0x03)
Data[3..6] = message length in bytes
Data[7.. +7] = utf-16 message bytes that you can easily parse as a string (little endian)

It seems to me that the packet buffer is always 1024 bytes and any data after your message is just old data that hasn't been overwritten. So you can just discard that in your parsing. Still checking about any crc stuff that should be in the message.

Edit: what seems weird to me is that your dump has the third byte as 0x01 but for me it is always 0x03, unsure what that is about

[gecko65]

@BeatSkip I also did identify
Data[0] Data[1] as &00F2 and Chat identifier
Data[2] == 0x03 I will check that on my device
Data[3] Data[4] I also have seen that as message size (2 bytes per UTF16-character")
Data[nn] Data[nn+1] is the UTF16-character

So far it seams my messages are cut, if the datagram exceeds 64 bytes. The longest messagte I was able to SEND as paylaod was 30 characters (in UTF-3 This are 60bytes).

[gecko65]

For a received message "789456123" I get
00 F2 01 00 00 00 14 37 00 38 00 39 00 34 00 35
00 36 00 31 00 32 00 33 00 00 00 00 88 00 00 00

[BeatSkip]

@gecko65 I have no such limitation, i can send a full string as long as i want. just tried to send over 2000 characters.
the first 508 are sent fine to the calc. and after 508 characters [note: that's around 1024-ish bytes of the max packet size] the text gets garbled due to a bug in my chunk splitting code (but that's just an error in my code).

and for return i can receive long strings fine too.

[gecko65]

Get Info from prime has the following format
00 FA 01 00 00 00 6C 00 00 00 00 00 00 00 00 00

Sending time and date has the following format
00 00 E7 01 00 00 00 0A 00 00 54 1E 16 02 0C 16

Get a screenshot begins with
00 FC 01 00 00 46 29 9C 9A 08 FF FF FF FF 89 50
Fails, because of expected 17942 bytes but only got 1134 bytes, output corrupted

Get a backup
00 F7 01 00 00 27 28 00 1E 05 AB 63 00 61 00 6C
Fails: packet out of sequence, got 0, expected 10

Example for send Key
00 00 EC 01 00 00 00 01 17
Need to check for Key-IDs ;-) Maybe in HPs Prime documenation

[gecko65]

@BeatSkip Which firmware version do you have?
My is 2.1.14538 (2020 12 17)
Hardwareversion: D
OS: V2.060.650

[BeatSkip]

you're out-of-date it seems

[debrouxl]

FWIW, back in the day, I could reliably transfer multi-kilobyte-sized variables. For a long time, TILP has unconditionally redirected terminal output to a file on Windows for the same reason as hplp, namely the slowness of the classic Windows terminal. With TI graphing calculators in USB vendor-specific class using USB bulk transfers, terminal slowness does not adversely affect transfer reliability; with the Prime in HID mode using USB interrupt transfers, it clearly did...

You can see occurrences of that 0x03 byte value in PR #4 .

[BeatSkip]

@debrouxl aha, so hplp does account for the 0x03, then it's just PrimeComm that's looking for the 0x01.
Currently i'm watching through a couple conference video's that reveal some source code of the Prime. and i'm starting to
create some up-to-date documentation of the prime protocol. as there's nothing up-to-date with more than the basics it seems.

but it seems that apart from my bug in packet splitting, i can just transfer multi-kilobyte packets without issue.

[adriweb]

Even though it's very old, maybe https://tiplanet.org/hpwiki/index.php?title=HP_Prime/Linking_Protocol could be updated?

[BeatSkip]

@adriweb yeah, from what I gathered around there's a tonne different with the modern protocol. That document is last updated in 2013

[metawops]

Even though it's very old, maybe https://tiplanet.org/hpwiki/index.php?title=HP_Prime/Linking_Protocol could be updated?

Oh, come on, please not on a TI website! 🤦‍♂️ There‘s plenty of HP calculators dedicated websites like hpcalc.org or hpmuseum.org (has a forum) or the Wiki at www.wiki4hp.com/doku.php?id=prime:start, to name just a few.

[adriweb]

Oh, come on, please not on a TI website! 🤦‍♂️ There‘s plenty of HP calculators dedicated websites like hpcalc.org or hpmuseum.org (has a forum) or the Wiki at www.wiki4hp.com/doku.php?id=prime:start, to name just a few.

When this was created, there was nowhere else to actively talk about the Prime, much less in a good wiki (mediawiki-based, anyway) way.
And it's not like the Prime community is very active, it's like 0.5% of what the TI community does, so.... But regardless, TI-Planet is not TI-only-focused, we literally deal with all brands, and the HP Prime won twice the back-to-school comparison guides, if you needed some more proof that it's not biased towards TI.

[BeatSkip]

@adriweb you are totally right, it seems to me that that little bit of community that the hp prime had is slowly bleeding to death. And honestly I think that is at least partially due to the fact there is so little and vague documentation and absolutely no good tooling to make the prime do what it's capable of. That's what I'm trying to fix with at least proper tooling. Hopefully it'll live up a bit when more people have easy access to their calculator. Just sad that the 'risk of influencing the examn mode' chokehold is limiting any progress, while I'm just toying with my hardware as an engineer.

[debrouxl]

Besides, that very wiki4hp page links to the TI-Planet hpwiki , both the root page and the USB protocol page :)

Your goal is laudable, but you probably know that you're facing a huge uphill battle creating significant activity for the Prime series' superior hardware and good, in some aspect superior software ;)

[BeatSkip]

@debrouxl oh yes, I know it'll be pretty much impossible to bring life into it. but als long as i see it as a fun and interesting challenge to make this tooling and some people appreciate the result. that's good enough for me :) even if it's just me appreciating a better tool, that's also fine haha

[BeatSkip]

@gecko65 what's your current status? Just got multi-part compressed messages via the newer protocol working! The newer protocol is mostly a wrapper around the old protocol with some additional checking and acknowledgements. Now working on the actual commands and features running on the protocol.

[BeatSkip]

@debrouxl if you are interested in maybe adding some newer features to your library, I can give you some explanations about the newer protocol. Still working out some final details on the CRC checking and I'll eventually document it. But that may take some time.

[metawops]

Guys, I was thinking ... wouldn't it be way cooler to have a VSCode Extension that allows us to freely write code for the HP Prime (and maybe even later other HP calcs, like the 48/49/50 series) with syntax highlighting and transferring it to/from the physical calculator? With enough effort I think even the full functionality of the HP Connectivity Kit could be integrated into VSCode. Just a proposal ...

[BeatSkip]

@metawops well, so that's pretty much what I'm going for. But then without vs code, no installation, no drivers, in your browser. Same text editor as vs code (Monaco editor also has a Web based version). Basically connectivity kit + ide with just opening a link. And native install with one click possible as pwa :)

[metawops]

@metawops well, so that's pretty much what I'm going for. But then without vs code, no installation, no drivers, in your browser. Same text editor as vs code (Monaco editor also has a Web based version). Basically connectivity kit + ide with just opening a link. And native install with one click possible as pwa :)

Yeah. But not inside VSCode which is THE code editor of tons of developers already. Plus, you have to code things like syntax highlighting yourself, from scratch. While in VSCode you'd define a language description and that's it. I have VSCode open all the time and when I decide to write a little HP Prime program I'd have to switch to the web browser and open a new URL.
Well, it was just a suggestion. I don't have the skills to write a VSCode extension. By all means, Oscar, do the thing you want to do! No problem! I was just suggesting. ;-)

[adriweb]

The thing is that when a stand-alone thing is ready and works as expected, then it can be used to make other things, including, why not, a component of an IDE extension, sure :)

[debrouxl]

Exactly. It may be possible, even easy, to shoehorn a library for WebUSB-based communication (preferably JS/TS for WebUSB glue code, and another programming language which can be targeted to WebAssembly) into a VSCode extension, but said standalone library needs to exist first.

Tons of other developers do also prefer code editors not based on heavyweight Web technologies :)

[BeatSkip]

@debrouxl you're right, it's trivial to port my code over to work on a native platform and make a VScode extension for example.
i've made an abstraction layer between my current protocol layer and the actual WebUsb stuff. so once this is done, i'd say within two hours the whole protocol library can be converted to run on windows and use native Hid libraries to communicate with the calculator. as the actual protocol library is purely written in C# (that is just so happen to compile to wasm atm, but no reason you can't just run it as normal). the library that couples my prime protocol library and the actual browser webusb stuff is actually 99% compatible with https://github.com/mikeobrien/HidLibrary that runs on windows. in a later version i'll make it 100% compatible so people can just choose how and where they want to run the library.

I'm planning on separating my web based code editor project and the actual prime communication library once i'm at a working alpha state. (they're already separate projects, just in the same git repo).

[debrouxl]

Nowadays, C# has better portability than it had in the mildly distant past, as long as one does not use OS-specific calls, obviously, be them for Windows or other OS.

And yeah, multiple projects in the same Git repo are easier to handle in the beginning than multiple repos and submodules, they can be split to separate repositories later as time permits and the rate of change decreases.

[BeatSkip]

@debrouxl yeah, I used to be full on c++ for platform flexibility and performance. But c# is going places and the ease is so convenient!

[BeatSkip]

Might I suggest moving this discussion over to the discussion board on my repo? As I do like the ideas and input. And I think the issue here has pretty much been solved right? Haha

BeatSkip/PrimeDev#2

[BeatSkip]

Although @debrouxl do you happen to know what's happening with the screenshot png header bug in the communication? From the de compilation of the connectivity kit I've extracted debug strings noting there is a bug and it seems the connectivity kit does something to fix the header (and colors). But do you know what?

[debrouxl]

All I might have known about screenshot-related weirdness is in libhpcalcs/src/prime_cmd.c::calc_prime_r_recv_screen() :)

[Cyrille-de-Brebisson]

Hello guys,

It looks like you are having some great fun :-) but you might save yourself some frustration getting info directly from the horse's mouth, so to speak!

I have not spent the time reading all the communication going on in the various threads, but if you have specific questions, I would be more than willing to help you answer them.

Cyrille

[BeatSkip]

Aha! great to have you here @Cyrille-de-Brebisson! Well, i happen to have a couple questions for you.

• When you request a screenshot. The colors are a mess, even when i litterally resend the same commands as the connectivity kit does. I presume the connectivity kit does some magic that corrects the header?

• when the conection is going i keep getting heartbeats (at least i presume) starting with FE 01 FF. I decode this as:
  FE -> out of band packet
  01 -> ACK
  FF -> should be acking the sequence number but FF is not valid sequence number.
  is this purely a heartbeat or is there some additional data appended?

When i request a backup Command 0xF9, i seem to receive a zlib encoded file containing a summary of the entire
contents. but only when i restart my program and reinitiate the connection i seem to get a contstant stream of file packets into my program. Oddly enough in the V1 protocol while i was working in the V2 protocol before restart. correctly ACK the last packet of the summary file (i think).

• I've seen a new command pass by in my communication dumps and can't seem to find out what it does. 0xFD, with a content length of 7 that always is: 0xFD 0x03 0x00 0x00 0x00 0x01 0x03

what is the proper order to request list of files at the beginning? currently i'm using:
1. initiate connection.
2. request check ready command
3. request calculator info
4. check build, if build >10500, ~~
5. send protocol change V2 command, note the last bit of the protocol change to 0x00, so V2 not V3
~~ (or new protocol that just seems to limit all the packet lengths to below 100 from the calculator somehow)
6. request backup. 0xF9 wrapped in V2 packet.
7. receive multi-packt message with summary.

but somehow all the actual files only come after restarting the program. but also when i restart the program
wand i don't want the files. as that sudden mountain of packets on restart before initialization f's with the
~~ program (although to be fair, that is my library just not handling it correctly yet)

edit: this turned out to be a bug in my back and forth acking with the protocol. now it dumps out everything as expected!

edit2: I can't seem to get the compression right. anything special going on? decompression of data dumps works fine, but actual compression of content doesn't seem to go happily. i'm using regular zlib deflate/inflate (with zlib header) to compress/decompress.

Note this is all with G2 calculator on build 14603

all the code is at:
https://github.com/BeatSkip/PrimeDev

note: still very much a mix and match between different libraries and sources, still dilligently rebuilding everything into a neat state, so code readability is horrible and jumping around between classes is still very much a thing, As I've only been working on this project for the last two weeks or so outside my day job as mech. engineer.

[Cyrille-de-Brebisson]

Hello,

This is all VERY old to me, so please accept my excuses if I mess up!

When you request a screenshot. The colors are a mess, even when i litterally resend the same commands as the connectivity kit does. I presume the connectivity kit does some magic that corrects the header?

Well, there are a LOT of issues in the whole screen send. But most of the issues come from the fact that multiple color schemes are used by various part of the SW. Some time it is RGB, sometimes BGR... Depending on the header type, it might be one or the other! sorry about that.

I attched some code for you to look at...
screen decoce.TXT

when the conection is going i keep getting heartbeats (at least i presume) starting with FE 01 FF. I decode this as:
FE -> out of band packet
01 -> ACK

FF -> should be acking the sequence number but FF is not valid sequence number.

This is the structure of said packet. Hope that this helps. apparently, the 3rd byte has no meaning on an ack...
struct TAckPacket {
u8 Byte254; // Packet ID: out of bounds: 254
u8 NewProtocolCommand; // command at 0: nack, 1: ack
u8 SequenceToResend; // no meaning for ack...
u8 unused_0;
u32 BlockPosition; // Position associated with first bit in this packet
u32 IOMessageID; // id of the IO Message beeing acked here...
void inline init(u8 Sequence, u32 position, u32 IOMessageID)
{ Byte254=254; NewProtocolCommand= 0; SequenceToResend= Sequence; unused_0= 0; BlockPosition= position; this->IOMessageID= IOMessageID; }
};

When i request a backup Command 0xF9, i seem to receive a zlib encoded file containing a summary of the entire contents. but only when i restart my program and reinitiate the connection i seem to get a contstant stream of file packets into my program. Oddly enough in the V1 protocol while i was working in the V2 protocol before restart. correctly ACK the last packet of the summary file (i think).

Here is the code that is a responce to the 0xf9 command:
bool CCalc::SendWholeCalc(u32 epid)
{
Calc->SaveCalcData(); //save our current stuff before sending, else

// send config settings

IOSendFile(epid, CALCSettings, OTNone, false, false);
IOSendFile(epid, CALCsettings, OTNone, false, false);
IOSendFile(epid, CASSettings, OTNone, false, false);
///send home variables
IOSendFile(epid, CALChpvars, OTNone, false, false);

wchar_t name[MaxAppNameSize]= L"\0\0";

//// Send List
name[0]=L'L'; name[2]=L'\0';
for (name[1]= L'0'; name[1]<=L'9'; name[1]++) IOSendFile(epid, name,OTLists,false,false);

//// Send Matrix
name[0]=L'M'; name[2]=L'\0';
for (name[1]= L'0'; name[1]<=L'9'; name[1]++) IOSendFile(epid, name,OTMatrices,false,false);

//// Send Programs
for (Int i=0; i<Programs.Size; i++) IOSendFile(epid, Programs[i]->GetFileName(),OTPrograms,false,false); //use this to send whole content

//// Send Notes
for (Int i=0; i<NoteList.Size; i++) IOSendFile(epid, NoteList[i].name,OTNotes,false,false);

// send Test Mode configurations
wchar_t n[CTestMode::MAX_SIZE];
for(Int i = 0; i < Exams.Size; i++) IOSendFile(epid, Exams[i]->get_name(n), OTExamModes, false, false); //send the exam mode configs

/// Send Apps
for(Int i=0; i<Apps.Size; i++) IOSendFile(epid, Calc->Apps[i]->GetFileName(), OTApps,false,false);

//// termination - this let's the CK know "end of data"
u8 b[6]= { IOPROT_GETCALC, GetIoProtVersionNum(), 0, 0, 0, 0 };
IOManager.PostMessage(TIOMessage::New(6, b), epid);
return true;
}

Strangely enough, I thought that it was changed to send a zip at some point... but I do not remember...

I've seen a new command pass by in my communication dumps and can't seem to find out what it does. 0xFD, with a content length of 7 that always is: 0xFD 0x03 0x00 0x00 0x00 0x01 0x03

#define IOPROT_GETIO_VERSION 253 // handle a return of multi byte IO protocol version packet information (switch to newer protocol or similar)
here is how the calc responds to it... Here it basically tells you: I can handle protocol version 3... to which the calc will respond I will switch to version ? (with the last byte being the version number)
if(cmd==IOPROT_GETIO_VERSION) //receiving response regarding IO protocol version
{
if(m->data()[0]<=GetIoProtVersionNum()) return true; // not capable of higher then we are set to at this moment?
//if we are currently using slower protocol, switch to highest supported
u8 ver= ::min(m->data()[0],(u8)IOPROTOCOL_VERSION_NUM);
TProtocolVersionRequestResponce p(ver); ep->PostMessage(TIOMessage::New(sizeof(p), (u8*)&p, 0)); SetIoProtVersionNum(ver); //set version number on this connection
return true;
}

what is the proper order to request list of files at the beginning? currently i'm using:

initiate connection.
request check ready command
request calculator info
check build, if build > ~10500,
send protocol change V2 command, note the last bit of the protocol change to 0x00, so V2 not V3 (or new protocol v2 as that just seems to limit all the packet lengths to below 100 from the calculator somehow)
request backup. 0xF9 wrapped in V2 packet.
receive multi-packt message with summary.

I do not remember, sorry!

but somehow all the actual files only come after restarting the program. but also when i restart the program and i don't want the files. as that sudden mountain of packets on restart before initialization f's with the program (although to be fair, that is my library just not handling it correctly yet)

I do have a flush on first connection from memory, specifically to handle that type of issues...

Keep the questions going, I will try to keep answering them...

Cyrille

[BeatSkip]

@Cyrille-de-Brebisson thanks for the info! Sure helps, if there is more I'll let you know. One question, do yo happen to know anything on the current state of development on the prime?
Is there still active development going on?
And is the python functionality going to be extended?
(LVGL compiled into the system? This would be sooo nice(!) as it would allow full freedom in app development while still keeping the examn mode intact and secure)

edit:
oh yeah, and the source code for the backup did help nicely in removing some guesswork when creating the backup procedure.

nicely working as you can see. the library switches all received files to a separate HpBackup class containing everything when it is in backup mode.

the backup mode now starts when it receives the actual ACK from the backup request and stops when the last backup done message comes in, just to make sure the backup is nice and complete.

[BeatSkip]

@Cyrille-de-Brebisson I've got all the communication working now and the code editor is implemented. I'm just having trouble decoding the apps. I've managed to reverse engineer where all the sections in the apps that are sent over are sliced. So I have separate files. But I'm struggling with the header contents of the 'hpapp' file and related specific files. Where the lengths of the filenames inside the programs are encoded etc. And what exactly I need to re-encode in the headers when I want to send back a modified app

[Cyrille-de-Brebisson]

Hello,

Well... problem here...
Basically, I do NOT know what the format is, because there is not really one...
Let me explain, file load/save is done through an abstraction system which "describes" an in memory structure as a series of items of various types, located at various places in memory, with various IDs.
It uses this to generate an output file which will have the id/type for each item.
Then, the loading "attempts" do to the reverse, potentially transforming input data from one format (in is 16 bit, but 32 bit is needed in RAM? no problem, it handles it for you!)...
Although this is great for my as it makes save/load much easier and allows updating from V1 to V2 easly... but not easy for someone like you who wants to reverse engineer the format!
Note that programs now use the same system to save all the programs data....

But if you want, indeed, to decode apps, it will be a lot of work!

For example, this describes the persistance stuff for the finance app...
if starts with a 32 bit number which is our flags. id=1
then a vector of 12HP_Reals (id= 2)...
etc...
now, when reading the file, the data can be in any order (thanks to the id), not everything has to be present. if id 2 in the input is a batch of 5 u32, they will be converted to 5 HP_Reals, and the others will we left to default...

const DATATYPE::Numb::TThisPersistedType DATATYPE::Numb::persisted_type =
{
PersistedType3(Numb, 0, 0, TPersistedTypeBase::f_UsePointerProvided, NULL, NULL, NULL),
{
Persisted(u32, 1, Numb, flags, 1),
Persisted(HP_Real, 2, Numb, TVM, 12),
Persisted(HP_Real, 3, Numb, IConv, 3),
Persisted(HP_Real, 4, Numb, DDAYS, 3),
PersistedS(MObj, 5, Numb::TCashFlow::persisted_type, Numb, CashFlow, 1), // cf!
Persisted(HP_Real, 6, Numb, DEPR, 6),
Persisted(HP_Real, 7, Numb, BRKEV, 5),
Persisted(HP_Real, 8, Numb, PChange, 8),
Persisted(HP_Real, 9, Numb, BOND, 9),
Persisted(HP_Real, 10, Numb, BS, 8),
},
{ &Numb::TCashFlow::persisted_type.base }
};

So, each "item" in the file starts with a 32 bit stuff which has that format...
// This first u32 is what actually gets saved in the file!!!!
u32 Type: 4, // Matches one of the TTypes above
TypeModifiers: 2, // See modifiers above
Compressed: 1, // set if data compressed
TypeId: 15, // Either matches the current structure TypeId, OR matches a sub-structure thereof. A sub structure will be loaded and placed as needed.
// Can also be set to 0 for facility. At this point, it is assumed to be current structure TypeId
MemberId: 10; // An id for yourself. Assign one per saved data member of your structure. Do not reuse if member is removed. This does provide an indirection from file to data structure so that
// you can load old/new files (what is not recognized does not get loaded)...
// DO NOT USE 0, THIS IS A PROHIBITED NUMBER!!! (they get ignored in order to make sure that if you mess up and do not define all the members, you do not end up with a number of 0 members at the end of your definition!)

Cyrille