Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Log parsing performance #41

Merged
merged 8 commits into from
Oct 16, 2022
Merged

Conversation

jack-ullery
Copy link
Owner

Description

Since there can be a large number of AppArmor logs on a system, we should prioritize performance when parsing them.

Previously, parsing the logs would be an O (n log(n) ) operation. These changes improve it to an O( n ) operation.

Change 1 - Do not store LogTableEntry structs in the database

Previously, we would store the LogTableEntry structs in the database, so that we could retrieve them when a log is selected. Adding a single log to the DB takes O ( log(n) ) time according to the implementation for std::map, where n is the number of logs with a constant number for AppArmor profiles. However, this was functionality was not used as of #39.

Instead of storing a LogTableEntry struct in the database for each profile, we instead store an integer for the number of Logs parsed for each Profile. Now, adding a single log takes constant time, assuming the number of Profiles is constant.

Change 2 - Use cursor when parsing logs

journalctl can provide a cursor when parsing logs on multiple calls. It will only print logs that occurred after the cursor, if you provide one. I made changes for a few classes to take advantage of this functionality.

Now ConsoleThread will remember the cursor every time it gets a batch of logs, and use that cursor the next time it queries for logs. This ensures that duplicate logs are do not returned by CommandCaller, which lowers the amount of logs parsed when refreshing tabs.

Change 3 - Update unit tests to reflect changes

These changes involved modifying Database, AppArmorCaller, and LogAdapter. The unit tests needed to be rewritten to reflect these changes. I also removed the DatabaseMock class, because it has not been necessary since #37.

@jack-ullery jack-ullery linked an issue Oct 16, 2022 that may be closed by this pull request
@jack-ullery jack-ullery merged commit 224c19a into main Oct 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Long load times for logs
1 participant