udffsck 1.00-beta

[argorain]

I just finished udffsck 1.00-beta. It is first open source implementation for Linux, which is capable of either check and fix UDF file system.

It is able to check and fixes UDF up to version 2.01, just like rest of package.

[pali]

You probably need to clear LDFLAGS for cmocka because cmocka is broken... I already wrote detail here: #7 (comment) (with simple patch).

tcc now compiled udftools successfully, but tests are failing... bug in code?

[pali]

You forgot to enable this -DWITH_STATIC_LIB=ON flag for other configurations which have --static flag in LDFLAGS. And so it failed on arm and ppc.

[argorain]

Ok, ARM compiles, unit tests are passing but then it get stuck on first real data.
TCC binary seems to be buggy, so there will be some investigation to do.

[pali]

ad ARM: maybe there is similar infinity loop like that with getchar()?
ad TCC: tests do not crash, but they just fail, right? Try to run that test under valgrind (both gcc and tcc version) and look if there is not something suspicious.

[argorain]

ad ARM: it should not. I guess there is something with access to data from qemu, but I'll check it.
ad TCC: yes, it fails, not crashes. I haven't used Valgrind before (altrough I used SonarQube). Any recommendations?

[pali]

open/close/read/write/mmap should work fine in qemu. IIRC there were only problems with threads and some synchronizations. Another reason can be slowness. travis automatically kills jobs when there is no ouput on stdout/stderr for 10 minutes. Try to add some debugging output to stdout/stderr...

For valgrind, just run valgrind your_executable params... and look on stderr. If there are any problems valgrind print them. By default it checks for memory corruptions (reading from uninitialized memory, writing behind end of allocated block, etc..) at process runtime by tracking every memory access. It emulates x86 instructions, therefore is slow too, so run it locally.

[argorain]

Yeah, it is killed because it stalled. I'll try it locally to see what happens.

Memory access was debugged using clang's address sanitizer, but I'll definatelly run Valgrind as well.

[argorain]

Very well, ARM cmocka tests went crazy. It seems that forking mechanism is not working properly for some reason. Good thing is it works (more or less) so I guess I can now consider CI as up and running for udffsck.

[pali]

If you have some ARM board (e.g. Raspberry PI) you can try to run ARM test natively. To check if problem is in udffsck code or in qemu-arm. Or you can install full ARM based linux distribution and run it under qemu-system-arm.

And for x86 (both gcc and tcc), you should run all tests under valgrind. Just call valgrind ./udffsck --your-fsck-params and check stderr output where valgrind put all messages (every line starts with == so you can detect if it comes from valgrind or from udffsck itself). Do that locally for both gcc and tcc builds.

[argorain]

Ad ARM: I tried that but I got stuck in some strange error because raspbian GCC is not able to build C99 sources if I remember correctly. I guess I'll end with qemu-system-arm.

Running all tests you mean running cmocka under valgrind, right?

[argorain]

Ad tcc: that thing is producing hard-to-debug code because it ignore -O flags (and has no equivalent to it) and for some reason even when built with -g, gdb cannot access locals, not even specified function because it is probably unrolled somewhere. What I am seeing, it reorganise code in way that sometimes it variable incremented one times more than it should (seen at for cycles for example, comparison between tcc and gcc/clang). And to be honest, I have no idea why and since gdb is useless, I don't know how to debug either. Any ideas on that matter?

[argorain]

Fyi, random increment is in file utils.c:209, variable 'line'. Needs to be built with -DHEXPRINT, run with this: ./udffsck ../../udf-samples/bs2048-r0201-dirty-file-tree.img -vvv

[pali]

Raspbian GCC must be able to compile C99 sources... I do not see reason why not. Otherwise tons of Debian packages would not be available on Raspberry PI, which is not truth. Do you see any error?

[KaliszAd]

What is the status of merging? This seems interesting (I guess this is related to this work: https://dspace.vutbr.cz/xmlui/bitstream/handle/11012/65230/vladyka.pdf?sequence=-1)

[argorain]

Hi, yes, that diploma thesis is mine. Current status is following: it seems to be working somehow on both x86 and x86-64 linux using GCC or LLVM. TCC is work-in-progress as well as ARM support. Big endian architectures are out-of-scope right now. Fsck itself can right now do checks and fixing for UDF 2.01 or older. Newer versions are not supported yet. Remember it is still unreleased and rather untested fsck so I would not recommend it for production grade systems, but feel free to test it right now because planned chenges are only for compatibility support or bugfixes at this moment.

[KaliszAd]

Thanks for the quick update. I found your work by looking how to leverage dm-integrity not only for checksum mismatch discovery but also for repair if the same block is to be found on a different device e.g. in a RAID mirror or parity setup.

I have a CD-image, that read correctly with dd, but the files were all unreadable. Do you think, such an image is a good test?

Keep up the good work!

[argorain]

I see.

Anyway, regarding CD, if it has UDF on it, it can be interesting test. My guess is that filesystem will be beyond repair, but you can definitely try that.

[Skirmisher]

Just chipping in my test results here.

I was testing a large UDF partition for Windows interoperability, which I placed my Fedora 28 home partition on. I used the format-udf tool to create the most compatible setup. Unfortunately, the Linux UDF driver seems to cause some issues that leads Windows to refuse to write to the partition, leaving it read-only after a couple days of use.

I decided to give this tool a go, since Windows chkdsk wasn't much help and I had no other options. Alas, all it does it complain about LVID timestamps and then segfault. It also crashes valgrind, but at least the output of that is more useful; it's attached below.

Running with any verbosity level above the default gives me infinite warnings of "Error marking block as used. It is already marked." for seemingly every block in the filesystem.

The partition in question no longer exists, as I've moved on to other options for multiplatform filesystems (and in fact am still looking for a usable solution). However, I'm happy to offer more help in testing, though my time may be limited in the future.

udffsck_valgrind.txt

[DanMan]

Would it be possible to release the checking part without the fixing part (if that's not ready yet), so you can at least see, if your partition is messed up?

I also gladly help testing it. I'm currently moving my games from a smaller disk to a bigger UDF formatted one. I'm already having problems with corrupted files and whatnot, so I'd make a good test animal. ;)

Oh, and whenever I let it "Fix SBD", it breaks the partition with:

munmap_chunk(): invalid pointer
[ERROR] Error identifiing FSD. Tag ID: 0xffff
[ERROR] Unable to continue without FSD. Consider submitting bug report. Exiting.

[Randrianasulu]

just something I found yesterday (not as big as this pull request but something)

https://github.com/smagnani/udftools/commits/for_pali