macOS Unified Access Logging (UAL)

[dabresua]

This PR completes the work done by @puffyCid in #7259 which also continued the work made by @GarretReece and @mike-myers-tob.

The funcionality enables osquery to read from UAL using its API.

My contribution is explained in the blueprint #7591 which, in summary:

• Limits the rows the table returns using a flag (ual_max_rows), 100 by default
• If the user launches a non-specific query select * from unified_log returns the rows continuing from the last one, so user can extract all the logs by querying multiple times.

Adds unitary tests for the flag and for the sequential extraction.

Closes:

• Limit macOS UAL table rows number and provide sequential extraction #7591
• Add macOS Unified Access Logging parser #7259
• Add a table for the unified log on darwin #6904
• Deprecation of ASL logging, and asl table #5760

Co-authored-by: puffycyan@gmail.com
Co-authored-by: GarretReece@users.noreply.github.com

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: GarretReece / name: Garret Reece (dda3ec3, e546021, add5666, d419987, 7fd19d7, 846b169, 38054f2, bd0240f, c48b54b, 0728e87, bc2040e, 7c454b2, fb826bf)
• ✅ login: mike-myers-tob / name: Mike Myers (84f71d0, 8dfbf6f, dfe2d9a)
• ✅ login: puffyCid (ccf4996, 19b5ac5)
• ✅ login: dabresua / name: Daniel Bretón (cc52835, 62f4755, aa049a3)

[dabresua]

The last changes implement what we discussed at the osquery office hours 2022-05-24.

Getting data sequentially is now triggered by the command select * from unified_log where timestamp > -1. I've tried to implement a sentinel value like LAST_TIMESTAMP, but I found the SQLite engine was not expecting a non-numerical value and rejected it, returning nothing.

Instead of a flag, a HIDDEN column is used for max_rows. The default value is 100, the way to modify it is by adding the clause max_rows = N. For example, select * from unified_logs where max_rows = 123 returns up to 123 rows.

User shall combine both clauses, i.e. select * from unified logs where timestamp > -1 and max_rows = 1000. This query returns the first 1000 log entries, and if we run it again, it will return the next 1000 log entries ordered by timestamp.

[dabresua]

Finishes the work done in #7259

[mike-myers-tob]

@directionless is going to re-review but it's moving to the 5.5 milestone

[directionless]

Code looks mostly fine to me. @alessandrogario @Smjert either you look?

One question -- I think this has a single tracking state for the last seen state. So if you had two queries each using this table and the -1 it wouldn't work. Correct?

[directionless]

Suggested change

"The condition \"max_rows = N\" sets the maximum number of rows returned (100 by default). "

[directionless]

I tried select timestamp, process, substr(message, 0, 200) from unified_log where max_rows = 50 and timestamp = -1; and it returned no data and mostly wedged.

And I realized an awkward spot... If the SQL has where timestamp = -1, than the table has to return that, else sqlite will filter out the data. This means would cause the timestamp data to be lost.

[directionless]

Ha. Correcting myself, where timestamp > -1 works correctly.

And where timestamp = -1 seems like a footgun we probably shouldn't try to fix?

[dabresua]

Ha. Correcting myself, where timestamp > -1 works correctly.

And where timestamp = -1 seems like a footgun we probably shouldn't try to fix?

As you said before, SQLite allows only the data with timestamp = -1, so it will be filtered. However, someway the UAL engine gets stuck, I'm adding an escape condition for that situation.

[directionless]

I think this is probably mergable? At least, I don't think I have other comments. @Smjert, @alessandrogario Either of you have comments?

@dabresua I don't know if it makes sense, you could consider tossing in the github Co-authored-by for the other folks. Or not.

[directionless]

Sounds like we all think this is approvable. wooo!

Thank you so much for sticking with it @dabresua it's a huge PR and improvement