Refactor UsnJrnl logic

[noxdafox]

The refactoring enables the support for Journal records of version 3
and 4.

The declarations have been all moved to tsk_ntfs.h as they are
internal for the moment.

A generic TSK_USN_RECORD structure has been introduced. It will rely
on a union to contain the version specific data.

The version specific data is contained in TSK_USN_V*_DATA structures.
As the V2 and V3 structures contain the same information, they've been
grouped into a single TSK_USN_V3_DATA to signify the highes supported
version.

The TSK_USN_RECORD_HEADER is now merged into the TSK_USN_RECORD and
can be used by the User to determine the union type (v2, v3, v4).

This simplifies the callback logic as it now receives a
TSK_USN_RECORD structure only.

Signed-off-by: Matteo Cafasso noxdafox@gmail.com

[bcarrier]

I'm a bit torn here about making this very NTFS-specific (and naming all of the methods with ntfs in them to make that clear such as TSK_FS_NTFS_USN_RECORD) versus more generic for future growth (such as TSK_FS_XYZ_RECORD). Though, I'm not yet sure for a good name for XYZ to make it clear how it is different from the file system journal.

Do you have thoughts @noxdafox?

[noxdafox]

With: The declarations have been all moved to tsk_ntfs.h as they are internal for the moment. I meant that it's temporary up until we don't find a good solution.

My current focus is to make usnjls feature complete. To do so, we need to add support for V3 and V4 records (the code is ready, waiting for this patch to be merged).

Once that is done we will have a complete picture of the data which is involved and this will help us better defining the final API layout.

Regarding the naming, a possible approach could consist in introducing a specific "namespace" for logs and journals. It would be cool if TSK could support more of these high level features such as the Windows Event Log, the SystemD journal and, of course, the low level FS journals (NTFS, ExtX, ...). The namespace could be something on this line:

• TSK_FS_JRNL_XYZ_RECORD
• TSK_FS_LOG_XYZ_ENTRY

As most (if not all) the entries are representing events, a common header could be used.

struct TSK_FS_LOG_RECORD { int type; uint32_t time_sec; uint32_t time_nsec; };

A union could follow containing the type specific data. Here we're getting pretty deep into hairy details though.

What do you think about it @bcarrier?

[noxdafox]

Added a small fix and a change.

When discussing with some user they told me they'd prefer to have human readable reason fields also in the mac output format. This because they usually feed such outputs to log2timeline and they'd prefer to have the change reasons visible.

Example:
2.0|80|421778-9|421777-10|121599624|1504800762.405466200|DATA_OVERWRITE DATA_EXTEND FILE_CREATE BASIC_INFO_CHANGE CLOSE|0|0|8224|hvax64.exe

Previously, the change reasons were packed in a UInt32.

[noxdafox]

@bcarrier any input on this?

[simsong]

@noxdafox - Hi. I'm trying to clear out the old PRs. Is this still relevant?

[simsong]

Hi. Please rebase, resolve conflicts, and make sure it passes all check. Ideally add a unit test. Thanks.

[noxdafox]

Hello,

this PR adds support for newer NTFS USN records. It is relevant as long as the community believes they are needed.

The PR stalled due to lack of feedback so I guess that should be resumed first. I can rebase if needed but will this track be resumed then?

[simsong]

Hi! We are addressing many of these old PRs. Do you have a test disk image that demonstrates this PR works?

[simsong]

@noxdafox — I'm coming back to this. Do you have a test disk image? @bcarrier had concerns about this. It's also the case that this now conflicts with the current baseline.

I'm willing to accept the change if we can get a test disk (or if we can add a test for this with our current test disks), but I really need you to resolve the conflicts.

[noxdafox]

Rebase done. I am unsure I can produce a test image for this. Is there documentation on how to do so?

[simsong]

Rebase done. I am unsure I can produce a test image for this. Is there documentation on how to do so?

A test image is a small image that demonstrates the problem that you have fixed. If you attach the disk image to this ticket, I'll add it.

However, we can't accept this PR until all of the checks pass without failing. It has to compile...