Same Day Filter With Grace Period

[Zalgo2462]

As described in issue #14, we need to assure RITA that when we "finish" writing to a database, we won't write any more records to that database. The issue is, IPFIX-RITA doesn't "finish" writing to a database. If a pair of flow records come in with closing timestamps aligned with a given day, the program will insert the matched pair of flows into that day's database no matter what.

In order to add the idea of "finishing" a database import, we will implement a Same Day Filter With a Grace Period.

This filter is applied to the output module in the IPFIX-RITA converter pipeline.

Filter algorithm:

• If the date of the closing timestamp of the session record is the current day, we insert the record into today's database.
• If the date of the closing timestamp of the session record is the previous day, and today's grace period has not elapsed, we insert the record into yesterday's database.
• Otherwise, we drop the session

The filter can be written in terms of arbitrary periods instead of being date aligned. For the more general algorithm, please see issue #14.

Currently, the grace period should be set to 4 minutes (5 minutes, but we short 1 minute to account for minor clock drift). Ideally, this grace period will be pushed back to 1 hour.

Related issues:

• Align output databases to system timezone
• Implement import_finished metadatabase field