Error when trying to convert fmp12 file

[amoe]

Hi,

Thanks for these tools!

Working with git master at commit 4acc820. When trying to convert a .fmp12 file to JSON, I get the following error:

amoe@cslp019129 $ ./fmp2json foo.fmp12 out.json
**** UNRECOGNIZED CODE 0xc0 @ [1189] *****
Error code: 11

What would this mean? I think the file is valid. I can't provide the file itself but I can provide a hex dump of regions of the file if you need it.

Cheers!

[amoe]

Oddly, when testing with the released version 0.1.0, I get a different error:

 **** UNRECOGNIZED CODE 0x70 @ [1570] *****
Error code: 11

[evanmiller]

Try the ./fmpdump program for some more context on where it's choking. I have made some small changes since the 0.1.0 release to support more byte codes but likely have some more that need support! A hex dump of the surrounding "block" (4096 byte sector) would certainly help - this will be labeled with something like [ BLOCK 32 ] in the fmpdump.

[amoe]

Here's a log that shows the output from fmpdump failing at block 7733.
https://gist.github.com/amoe/ef256e1f88de4f7c62b545600126a076

Here are some dumps I tried to create but I'm not sure if I made them correctly --

amoe@cslp019129 $ xxd -s $((4096 * 7732)) -l 4096  file.fmp12 > block7732.dump
amoe@cslp019129 $ xxd -s $((4096 * 7733)) -l 4096  file.fmp12 > block7733.dump
amoe@cslp019129 $ xxd -s $((4096 * 7734)) -l 4096  file.fmp12 > block7734.dump

https://gist.github.com/amoe/52309f952eccc6762ab4b9ed5ea659fc

[evanmiller]

@amoe One last thing that would help is uncommenting these lines:

fmptools/src/fmp.c

Lines 262 to 267 in 4acc820

	/*
	fprintf(stderr, "ERROR processing block, reporting partial results...\n");
	block->this_id = next_block;
	if (!handle_block || handle_block(block, user_ctx))
	process_chunk_chain(file, block->chunk, handle_chunk, user_ctx);
	*/

Then recompile and re-run fmpdump. That will give me some more context about the error, which appears to be in block 7734.

[amoe]

@evanmiller
I tried to create this dump, see it here: https://gist.github.com/amoe/61bbc43b259e7e69bc3a291a5ee8707b

Created with ./fmpdump file.fmp12 >~/fmpdump2.log 2>&1, and I snipped stuff before block 7733

[evanmiller]

Notes for myself:

The problematic line is in block 7734:

01e364b0: b6ff 0011 c005 4b00 10c0 347d 4020 6580

0x11 is a data_simple of length three. The next code is 0x00. This is being interpreted as a data_simple of length 1 (consuming the 0x10 byte), but probably should be treated as a no-op instead. This would allow the 0x10 code to consume the next three bytes and reach the 0x40 "pop" code.

[evanmiller]

@amoe please see if the code in this branch fixes the issue for you:

https://github.com/evanmiller/fmptools/tree/issue-3

[amoe]

Thanks @evanmiller.
It did fix the issue with that block! fmpdump now runs for a lot longer. It bails out later at the boundary near block 94189.

out.log

[evanmiller]

@amoe Good to hear - I'll need a hex dump of that last block (94189) to make any further progress.

It also looks like some of the path integers are being incorrectly interpreted, but we can deal with that once the basic file conversion is working.

[amoe]

Here's a dump of 94189: block94189.txt

[evanmiller]

@amoe Got it. Try pulling the latest code from the issue-3 branch and see if that moves things further along.

[amoe]

@evanmiller Great stuff! fmpdump and fmp2json are now finishing successfully and dumping the whole file :)
Thanks!

[evanmiller]

Excellent! I'm going to merge in the changes and close this issue. If you encounter any data errors or problems with other files, please open a new issue.