Repository navigation
ActorMonitor
Text compliant with Leucine version 0.7.x
The behavior in small actor systems is usually simple enough
to predict. However, as systems get larger, and have longer
run times, this is less the case. There are two main reasons
for the application designer to want to have more insight in
what is going on under the hood. The first, usually during
design and testing is detailed debug information. What
letters are send where, in which order etc. The second,
usually during operations is runtime information. Which
actors are very busy, maybe overloaded, how many messages
are processed, are there any messages lost etc. To obtain
this information the GlobalMonitor is present.
We already saw
we need to mixin MonitorAid to be able go make use of the monitor capabilities.
There we used an instance of LocalMonitor
to keep the information local. Here we discuss
to possibility to monitor a larger (or all) your actors
with the GlobalMonitor. Both LocalMonitor and GlobalMonitor
implement the trait ActorMonitor.
Note that you may very well define several different instances
of your own Monitor which extend GlobalMonitor
and use them on different actor groups. Here we will limit
ourselves to one instance only. Contrary to the LocalMonitor,
instances of the GlobalMonitor are designed to be used for
multiple actors.
The GlobalMonitor is an abstract class you must override with your own
implementation, contrary to the LocalMonitor that you can use directly.
Here the probeInterval is a field you must implement by overriding. The
same holds true for tracing:
/**
* This is the public setting of tracing. Every actor has it personal setting as well.
* If tracing is active for this actor depends on both settings, in a symmetric manner.
* There are two levels of tracing. TraceFull and TraceCount. TraceFull traces every
* message with timestamp and action. This may be memory intensive. TraceCount just
* counts the messages from sender to receiver per letter. Usually this number is limited.
* However, both forms should only be used at debugging, since it requires a lot of
* synchronized operations.
* If both are Enabled or one is Enabled and the other is Default, the TraceFull is
* active. In all other cases it is not. TraceCount is active when both are at Default
* or one of the settings is Enabled. This implies that you can enable/disable
* the tracing TraceFull here as long as the personal tracing is Default or Enabled.
* Setting this to Disabled will always prohibit tracing TraceFull of this actor,
* and setting it to Default/Enabled leaves the fate in the hands of the personal setting.
* The personal setting defaults to Tracing.Default. You must define this setting in
* your monitor. If unsure what to do, try Tracing.Disabled first. */
def tracing: Tracing
/**
* Default probe interval. Set this to a reasonable value, say 5 seconds for short running
* applications and maybe 1 minute for servers. */
def probeInterval: FiniteDurationHere tracing does not have a default implementation, so you must choose one of
Tracing.Disabled, Tracing.Default or Tracing.Enabled, and the probeInterval.
Both fields may be changed, if you want, during usage. This can be handy to
zoom in on a particular problem you want to isolate.
The minimal implementation could look something like this:
/* If you want to use the monitor, you must implement some callback methods. Here we keep
* that simple and only report the after the application has completed. In bigger applications
* you might need to export the situation from time to time and purge the monitor to prevent
* data structures from getting to large. */
object Monitor extends GlobalMonitor :
import MonitorAid.{Trace, Action, Tracing}
/* Custom method to show the results obtained so far. Since this example only has one short
* run the results are show at the end. */
def show(samples: Boolean = false, details: Boolean = false): Unit = report(println,samples,details,details,details)
/* This is a short running application, probe often */
def probeInterval: FiniteDuration = 5.seconds
/* Global setting of tracing. Here we enable it for all actors. Since the personal setting is Default
* this should activate TraceCount and TraceFull. */
def tracing = Tracing.EnabledHere we make use of a build in report method that does some basic displaying for us. This report method has the following signature:
/**
* Create a full report to a given string writer. Contains only the present data. Note, since this
* action is performed outside the the probe loop, data may not be entirely consistent. If this is
* needed, act upon the Trigger callbacks. This is typically used for small actor systems at the
* end of the applications lifetime. */
def report(writeln: String => Unit, showSamples: Boolean, showPosts: Boolean, showTraces: Boolean, showLogs: Boolean): Unit =so with a few switches we can get a result quickly, without having to implement many
callback functions. The output is routed to the writeln function, which we choose
to be println in our extension Monitor. The show method is only there to
serve as an example. In larger systems, you may want to register separate
callback methods that are also present in GlobalMonitor:
/**
* Add a new trigger to the trigger event list. When new probe data on the system becomes available
* all triggers are called in random order. Each trigger receives the same copy of the Probed,
* containing the latest snapshot of data. Each trigger may return one or more purge flags to
* cleanup the collected data in the monitor. The combination of these flags (or-ed) is applied.
* If tracing is active, it is advised to at least clear all tracing each time. */
def register(trigger: Trigger): Unit
/** Remove one or more triggers from the trigger set. */
def remove(trigger: Trigger *): Unit
/** Remove all triggers from the trigger set. */
def removeAll(): Unitwhere the signature of Trigger is:
object GlobalMonitor :
/**
* Type used for triggers. Each trigger receives a snapshot of all collected data
* and must return its intentions for clearing that data. */
type Trigger = Probed => Iterable[Purge]
/** Case class returned to you which contains a snapshot of all collected data. */
case class Probed(samples: SortedMap[String,ActorMonitor.Record], posts: SortedMap[Actor.Post,Long], traces: SortedSet[MonitorAid.Trace])
/** Possible cleaning operations after processing the trigger. */
enum Purge :
/* Clears all actors from the samples list active or not. Active actors will quickly be reloaded. */
case Samples
/* Clears all inactive actors from the samples list. This resets the incarnation counter on non worker actors. */
case Inactive
/* Clears all inactive workers from the samples list. Since workers are never reused, their number can increase endlessly. */
case Workers
/* Clear all posts. A post is a counter for a message. The number of different posts usually does not increase endlessly, the counters do. */
case Posts
/* Clear all traces. A trace reports a certain action in the system. The number of traces keeps increasing. */
case TracesIt is possible to register multiple triggers as to process different pieces of information in different
parts of the code. All trigger callbacks receive exactly the same information and clearing is done
at the very end, based on the returned Purge of each call. It is advisable to at least clear
the traces if they are collected, for their number grows fast, and may lead to an out-of-memory
exception in the end.
In this simple example we explain how the monitor defined above can be used to
inspect the actor
we defined in the FamilyAid example. We extend it with:
class Tree(name: String, val parent: Option[Tree]) extends AcceptActor(Tree,name), FamilyTree[Tree], MonitorAid(Monitor) :
/* Show when the actor stops. */
final protected override def stopped(cause: Actor.Stop, complete: Boolean) =
/* This is executed when the root actor stops, which is at the end. */
if parent.isEmpty then monitor.show(true,true)manually stop it in main:
object Main :
... /* as in the original */
def main(args: Array[String]): Unit =
... /* as in the original */
/* Manually stop the actor. */
tree.stop(Actor.Stop.Finish)and the rest can remain as is. When we run we get the following result (beautified):
Created F0
Created F0.F1
Created F0.F2
Created F0.F1.F1
Created F0.F2.F1
Created F0.F1.F2
Created F0.F2.F2
Created F0.F2.F1.F1
Created F0.F2.F1.F2
Created F0.F2.F2.F1
Created F0.F1.F1.F1
Created F0.F1.F1.F2
Created F0.F2.F2.F2
Created F0.F1.F2.F1
Created F0.F1.F2.F2
=>> F0
=>> F0.F1
=>> F0.F2
=>> F0.F1.F1
=>> F0.F2.F1
=>> F0.F1.F2
=>> F0.F2.F2
=>> F0.F1.F1.F1
=>> F0.F2.F1.F1
=>> F0.F2.F2.F1
=>> F0.F1.F1.F2
=>> F0.F1.F2.F1
=>> F0.F2.F2.F2
=>> F0.F2.F1.F2
=>> F0.F1.F2.F2
<<= F0.F1.F1
<<= F0.F2.F2
<<= F0.F2.F1
<<= F0.F1.F2
<<= F0.F2.F2
<<= F0.F1.F1
<<= F0.F1
<<= F0.F2.F1
<<= F0.F2
<<= F0
<<= F0.F2
<<= F0.F1.F2
<<= F0
<<= F0.F1
<<= F0.F2
<<= F0
<<= F0.F2
<<= F0.F1
<<= F0
<<= F0.F1
<<= F0
<<= F0
<<= F0
<<= F0
[success] Total time: 1 s, completed 8 Jun 2023, 15:53:51
Called show()
All Samples:
'F0': incarnations=1, active=false; phase=Stop, stop=Finish, lettersSum=10, lettersMax=4, exceptionsSum=0, failedSum=0, needles=0, processLoad=13325ppm; namedChildrenNow=2, allChildrenNow=2, workersSum=0
'F0.F1': incarnations=1, active=false; phase=Stop, stop=Finish, lettersSum=6, lettersMax=2, exceptionsSum=0, failedSum=0, needles=0, processLoad=11437ppm; namedChildrenNow=2, allChildrenNow=2, workersSum=0
'F0.F1.F1': incarnations=1, active=false; phase=Stop, stop=Finish, lettersSum=4, lettersMax=2, exceptionsSum=0, failedSum=0, needles=0, processLoad=8351ppm; namedChildrenNow=2, allChildrenNow=2, workersSum=0
'F0.F1.F1.F1': incarnations=1, active=false; phase=Stop, stop=Finish, lettersSum=1, lettersMax=1, exceptionsSum=0, failedSum=0, needles=0, processLoad=4037ppm; namedChildrenNow=0, allChildrenNow=0, workersSum=0
'F0.F1.F1.F2': incarnations=1, active=false; phase=Stop, stop=Finish, lettersSum=1, lettersMax=1, exceptionsSum=0, failedSum=0, needles=0, processLoad=2655ppm; namedChildrenNow=0, allChildrenNow=0, workersSum=0
'F0.F1.F2': incarnations=1, active=false; phase=Stop, stop=Finish, lettersSum=4, lettersMax=2, exceptionsSum=0, failedSum=0, needles=0, processLoad=8709ppm; namedChildrenNow=2, allChildrenNow=2, workersSum=0
'F0.F1.F2.F1': incarnations=1, active=false; phase=Stop, stop=Finish, lettersSum=1, lettersMax=1, exceptionsSum=0, failedSum=0, needles=0, processLoad=2854ppm; namedChildrenNow=0, allChildrenNow=0, workersSum=0
'F0.F1.F2.F2': incarnations=1, active=false; phase=Stop, stop=Finish, lettersSum=1, lettersMax=1, exceptionsSum=0, failedSum=0, needles=0, processLoad=1163ppm; namedChildrenNow=0, allChildrenNow=0, workersSum=0
'F0.F2': incarnations=1, active=false; phase=Stop, stop=Finish, lettersSum=6, lettersMax=2, exceptionsSum=0, failedSum=0, needles=0, processLoad=7183ppm; namedChildrenNow=2, allChildrenNow=2, workersSum=0
'F0.F2.F1': incarnations=1, active=false; phase=Stop, stop=Finish, lettersSum=4, lettersMax=2, exceptionsSum=0, failedSum=0, needles=0, processLoad=7562ppm; namedChildrenNow=2, allChildrenNow=2, workersSum=0
'F0.F2.F1.F1': incarnations=1, active=false; phase=Stop, stop=Finish, lettersSum=1, lettersMax=1, exceptionsSum=0, failedSum=0, needles=0, processLoad=3669ppm; namedChildrenNow=0, allChildrenNow=0, workersSum=0
'F0.F2.F1.F2': incarnations=1, active=false; phase=Stop, stop=Finish, lettersSum=1, lettersMax=1, exceptionsSum=0, failedSum=0, needles=0, processLoad=798ppm; namedChildrenNow=0, allChildrenNow=0, workersSum=0
'F0.F2.F2': incarnations=1, active=false; phase=Stop, stop=Finish, lettersSum=4, lettersMax=2, exceptionsSum=0, failedSum=0, needles=0, processLoad=8440ppm; namedChildrenNow=2, allChildrenNow=2, workersSum=0
'F0.F2.F2.F1': incarnations=1, active=false; phase=Stop, stop=Finish, lettersSum=1, lettersMax=1, exceptionsSum=0, failedSum=0, needles=0, processLoad=3140ppm; namedChildrenNow=0, allChildrenNow=0, workersSum=0
'F0.F2.F2.F2': incarnations=1, active=false; phase=Stop, stop=Finish, lettersSum=1, lettersMax=1, exceptionsSum=0, failedSum=0, needles=0, processLoad=2270ppm; namedChildrenNow=0, allChildrenNow=0, workersSum=0
All Posts:
from=:x, to=F0, letter=Create(2,3): 1
from=:x, to=F0, letter=Forward: 1
from=:x, to=F0.F1, letter=Create(2,2): 1
from=:x, to=F0.F1, letter=Forward: 1
from=:x, to=F0.F1.F1, letter=Create(2,1): 1
from=:x, to=F0.F1.F1, letter=Forward: 1
from=:x, to=F0.F1.F1.F1, letter=Forward: 1
from=:x, to=F0.F1.F1.F2, letter=Forward: 1
from=:x, to=F0.F1.F2, letter=Create(2,1): 1
from=:x, to=F0.F1.F2, letter=Forward: 1
from=:x, to=F0.F1.F2.F1, letter=Forward: 1
from=:x, to=F0.F1.F2.F2, letter=Forward: 1
from=:x, to=F0.F2, letter=Create(2,2): 1
from=:x, to=F0.F2, letter=Forward: 1
from=:x, to=F0.F2.F1, letter=Create(2,1): 1
from=:x, to=F0.F2.F1, letter=Forward: 1
from=:x, to=F0.F2.F1.F1, letter=Forward: 1
from=:x, to=F0.F2.F1.F2, letter=Forward: 1
from=:x, to=F0.F2.F2, letter=Create(2,1): 1
from=:x, to=F0.F2.F2, letter=Forward: 1
from=:x, to=F0.F2.F2.F1, letter=Forward: 1
from=:x, to=F0.F2.F2.F2, letter=Forward: 1
from=F0.F1, to=F0, letter=Backward: 4
from=F0.F1.F1, to=F0.F1, letter=Backward: 2
from=F0.F1.F1.F1, to=F0.F1.F1, letter=Backward: 1
from=F0.F1.F1.F2, to=F0.F1.F1, letter=Backward: 1
from=F0.F1.F2, to=F0.F1, letter=Backward: 2
from=F0.F1.F2.F1, to=F0.F1.F2, letter=Backward: 1
from=F0.F1.F2.F2, to=F0.F1.F2, letter=Backward: 1
from=F0.F2, to=F0, letter=Backward: 4
from=F0.F2.F1, to=F0.F2, letter=Backward: 2
from=F0.F2.F1.F1, to=F0.F2.F1, letter=Backward: 1
from=F0.F2.F1.F2, to=F0.F2.F1, letter=Backward: 1
from=F0.F2.F2, to=F0.F2, letter=Backward: 2
from=F0.F2.F2.F1, to=F0.F2.F2, letter=Backward: 1
from=F0.F2.F2.F2, to=F0.F2.F2, letter=Backward: 1
All Traces:
time=8012441, nr=1, action=Created, actor=F0
time=15789897, nr=2, action=Accepted, from=:x, to=F0, mail=Received, letter=Create(2,3)
time=16222176, nr=3, action=Initiated, from=:x, to=F0, mail=Received, letter=Create(2,3)
time=17017736, nr=4, action=Created, actor=F0.F1
time=17821145, nr=5, action=Created, actor=F0.F2
time=19438072, nr=6, action=Accepted, from=:x, to=F0.F1, mail=Received, letter=Create(2,2)
time=19945136, nr=7, action=Accepted, from=:x, to=F0.F2, mail=Received, letter=Create(2,2)
time=20055104, nr=8, action=Initiated, from=:x, to=F0.F1, mail=Received, letter=Create(2,2)
time=20140448, nr=9, action=Created, actor=F0.F1.F1
time=20236983, nr=10, action=Initiated, from=:x, to=F0.F2, mail=Received, letter=Create(2,2)
time=20336196, nr=11, action=Created, actor=F0.F2.F1
time=20440159, nr=12, action=Created, actor=F0.F1.F2
time=20481056, nr=13, action=Completed, from=:x, to=F0, mail=Processed, letter=Create(2,3)
time=20587511, nr=14, action=Created, actor=F0.F2.F2
time=21249918, nr=15, action=Accepted, from=:x, to=F0.F2.F1, mail=Received, letter=Create(2,1)
time=21311346, nr=17, action=Accepted, from=:x, to=F0.F2.F2, mail=Received, letter=Create(2,1)
time=21321159, nr=16, action=Initiated, from=:x, to=F0.F2.F1, mail=Received, letter=Create(2,1)
time=21388024, nr=18, action=Created, actor=F0.F2.F1.F1
time=21471890, nr=19, action=Accepted, from=:x, to=F0.F1.F1, mail=Received, letter=Create(2,1)
time=21520738, nr=21, action=Initiated, from=:x, to=F0.F1.F1, mail=Received, letter=Create(2,1)
time=21525492, nr=20, action=Accepted, from=:x, to=F0.F1.F2, mail=Received, letter=Create(2,1)
time=21590368, nr=22, action=Created, actor=F0.F1.F1.F1
time=21597487, nr=23, action=Created, actor=F0.F2.F1.F2
time=21643227, nr=24, action=Completed, from=:x, to=F0.F2, mail=Processed, letter=Create(2,2)
time=21999000, nr=25, action=Initiated, from=:x, to=F0.F2.F2, mail=Received, letter=Create(2,1)
time=22020315, nr=26, action=Completed, from=:x, to=F0.F2.F1, mail=Processed, letter=Create(2,1)
time=22069574, nr=27, action=Created, actor=F0.F2.F2.F1
time=22290143, nr=28, action=Created, actor=F0.F2.F2.F2
time=22408426, nr=29, action=Created, actor=F0.F1.F1.F2
time=23141686, nr=30, action=Completed, from=:x, to=F0.F1, mail=Processed, letter=Create(2,2)
time=23245251, nr=31, action=Initiated, from=:x, to=F0.F1.F2, mail=Received, letter=Create(2,1)
time=23303543, nr=32, action=Created, actor=F0.F1.F2.F1
time=23312869, nr=33, action=Completed, from=:x, to=F0.F1.F1, mail=Processed, letter=Create(2,1)
time=24022905, nr=34, action=Completed, from=:x, to=F0.F2.F2, mail=Processed, letter=Create(2,1)
time=24114513, nr=35, action=Created, actor=F0.F1.F2.F2
time=24440722, nr=36, action=Completed, from=:x, to=F0.F1.F2, mail=Processed, letter=Create(2,1)
time=218384846, nr=37, action=Accepted, from=:x, to=F0, mail=Received, letter=Forward
time=218705141, nr=38, action=Initiated, from=:x, to=F0, mail=Received, letter=Forward
time=218880413, nr=39, action=Accepted, from=:x, to=F0.F1, mail=Received, letter=Forward
time=219034750, nr=40, action=Accepted, from=:x, to=F0.F2, mail=Received, letter=Forward
time=219132243, nr=41, action=Initiated, from=:x, to=F0.F1, mail=Received, letter=Forward
time=219157276, nr=42, action=Completed, from=:x, to=F0, mail=Processed, letter=Forward
time=219247360, nr=43, action=Accepted, from=:x, to=F0.F1.F1, mail=Received, letter=Forward
time=219254799, nr=44, action=Initiated, from=:x, to=F0.F2, mail=Received, letter=Forward
time=219373806, nr=45, action=Accepted, from=:x, to=F0.F2.F1, mail=Received, letter=Forward
time=219418478, nr=46, action=Accepted, from=:x, to=F0.F1.F2, mail=Received, letter=Forward
time=219537320, nr=47, action=Initiated, from=:x, to=F0.F1.F1, mail=Received, letter=Forward
time=219601495, nr=48, action=Accepted, from=:x, to=F0.F2.F2, mail=Received, letter=Forward
time=219644994, nr=49, action=Accepted, from=:x, to=F0.F1.F1.F1, mail=Received, letter=Forward
time=219667142, nr=50, action=Initiated, from=:x, to=F0.F2.F1, mail=Received, letter=Forward
time=219702243, nr=51, action=Completed, from=:x, to=F0.F1, mail=Processed, letter=Forward
time=219785730, nr=52, action=Accepted, from=:x, to=F0.F2.F1.F1, mail=Received, letter=Forward
time=219809157, nr=53, action=Initiated, from=:x, to=F0.F1.F2, mail=Received, letter=Forward
time=219820027, nr=54, action=Completed, from=:x, to=F0.F2, mail=Processed, letter=Forward
time=219894964, nr=55, action=Initiated, from=:x, to=F0.F2.F2, mail=Received, letter=Forward
time=219924145, nr=56, action=Accepted, from=:x, to=F0.F1.F1.F2, mail=Received, letter=Forward
time=219933427, nr=57, action=Accepted, from=:x, to=F0.F1.F2.F1, mail=Received, letter=Forward
time=219997007, nr=58, action=Accepted, from=:x, to=F0.F2.F2.F1, mail=Received, letter=Forward
time=220011793, nr=59, action=Initiated, from=:x, to=F0.F1.F1.F1, mail=Received, letter=Forward
time=220086224, nr=60, action=Accepted, from=:x, to=F0.F2.F1.F2, mail=Received, letter=Forward
time=220196811, nr=61, action=Initiated, from=:x, to=F0.F2.F1.F1, mail=Received, letter=Forward
time=220221714, nr=62, action=Accepted, from=:x, to=F0.F2.F2.F2, mail=Received, letter=Forward
time=220340731, nr=63, action=Initiated, from=:x, to=F0.F2.F2.F1, mail=Received, letter=Forward
time=220371856, nr=64, action=Completed, from=:x, to=F0.F1.F1, mail=Processed, letter=Forward
time=220531900, nr=65, action=Initiated, from=:x, to=F0.F1.F1.F2, mail=Received, letter=Forward
time=220536748, nr=66, action=Accepted, from=:x, to=F0.F1.F2.F2, mail=Received, letter=Forward
time=220632749, nr=67, action=Initiated, from=:x, to=F0.F1.F2.F1, mail=Received, letter=Forward
time=220653882, nr=68, action=Completed, from=:x, to=F0.F2.F2, mail=Processed, letter=Forward
time=220744523, nr=69, action=Initiated, from=:x, to=F0.F2.F2.F2, mail=Received, letter=Forward
time=221145924, nr=70, action=Completed, from=:x, to=F0.F1.F2, mail=Processed, letter=Forward
time=221149468, nr=71, action=Completed, from=:x, to=F0.F2.F1, mail=Processed, letter=Forward
time=221284703, nr=72, action=Initiated, from=:x, to=F0.F2.F1.F2, mail=Received, letter=Forward
time=221365131, nr=73, action=Initiated, from=:x, to=F0.F1.F2.F2, mail=Received, letter=Forward
time=221527520, nr=74, action=Accepted, from=F0.F1.F2.F1, to=F0.F1.F2, mail=Received, letter=Backward
time=221535704, nr=75, action=Accepted, from=F0.F2.F2.F1, to=F0.F2.F2, mail=Received, letter=Backward
time=221538108, nr=76, action=Accepted, from=F0.F1.F1.F2, to=F0.F1.F1, mail=Received, letter=Backward
time=221556884, nr=77, action=Accepted, from=F0.F2.F1.F2, to=F0.F2.F1, mail=Received, letter=Backward
time=221591093, nr=83, action=Completed, from=:x, to=F0.F1.F1.F2, mail=Processed, letter=Forward
time=221593980, nr=79, action=Completed, from=:x, to=F0.F2.F2.F1, mail=Processed, letter=Forward
time=221594892, nr=78, action=Accepted, from=F0.F1.F1.F1, to=F0.F1.F1, mail=Received, letter=Backward
time=221603690, nr=80, action=Completed, from=:x, to=F0.F2.F1.F2, mail=Processed, letter=Forward
time=221614310, nr=81, action=Accepted, from=F0.F2.F2.F2, to=F0.F2.F2, mail=Received, letter=Backward
time=221620088, nr=82, action=Accepted, from=F0.F2.F1.F1, to=F0.F2.F1, mail=Received, letter=Backward
time=221624329, nr=84, action=Completed, from=:x, to=F0.F1.F1.F1, mail=Processed, letter=Forward
time=221649907, nr=85, action=Completed, from=:x, to=F0.F2.F2.F2, mail=Processed, letter=Forward
time=221650638, nr=86, action=Initiated, from=F0.F1.F1.F2, to=F0.F1.F1, mail=Received, letter=Backward
time=221659795, nr=87, action=Initiated, from=F0.F2.F2.F1, to=F0.F2.F2, mail=Received, letter=Backward
time=221663271, nr=88, action=Completed, from=:x, to=F0.F2.F1.F1, mail=Processed, letter=Forward
time=221679526, nr=89, action=Initiated, from=F0.F2.F1.F2, to=F0.F2.F1, mail=Received, letter=Backward
time=221767775, nr=90, action=Completed, from=:x, to=F0.F1.F2.F1, mail=Processed, letter=Forward
time=221797288, nr=91, action=Accepted, from=F0.F1.F2.F2, to=F0.F1.F2, mail=Received, letter=Backward
time=221827149, nr=92, action=Completed, from=:x, to=F0.F1.F2.F2, mail=Processed, letter=Forward
time=221904439, nr=93, action=Initiated, from=F0.F1.F2.F1, to=F0.F1.F2, mail=Received, letter=Backward
time=222092991, nr=94, action=Accepted, from=F0.F2.F2, to=F0.F2, mail=Received, letter=Backward
time=222143036, nr=95, action=Completed, from=F0.F2.F2.F1, to=F0.F2.F2, mail=Processed, letter=Backward
time=222154291, nr=96, action=Accepted, from=F0.F2.F1, to=F0.F2, mail=Received, letter=Backward
time=222161772, nr=97, action=Initiated, from=F0.F2.F2.F2, to=F0.F2.F2, mail=Received, letter=Backward
time=222175627, nr=98, action=Accepted, from=F0.F1.F1, to=F0.F1, mail=Received, letter=Backward
time=222178332, nr=99, action=Completed, from=F0.F2.F1.F2, to=F0.F2.F1, mail=Processed, letter=Backward
time=222185196, nr=100, action=Initiated, from=F0.F2.F2, to=F0.F2, mail=Received, letter=Backward
time=222197444, nr=101, action=Initiated, from=F0.F2.F1.F1, to=F0.F2.F1, mail=Received, letter=Backward
time=222215338, nr=102, action=Completed, from=F0.F1.F1.F2, to=F0.F1.F1, mail=Processed, letter=Backward
time=222234837, nr=103, action=Initiated, from=F0.F1.F1.F1, to=F0.F1.F1, mail=Received, letter=Backward
time=222255781, nr=105, action=Accepted, from=F0.F1.F2, to=F0.F1, mail=Received, letter=Backward
time=222256404, nr=104, action=Accepted, from=F0.F2.F2, to=F0.F2, mail=Received, letter=Backward
time=222275178, nr=107, action=Completed, from=F0.F2.F2.F2, to=F0.F2.F2, mail=Processed, letter=Backward
time=222281066, nr=106, action=Completed, from=F0.F1.F2.F1, to=F0.F1.F2, mail=Processed, letter=Backward
time=222305979, nr=108, action=Initiated, from=F0.F1.F2.F2, to=F0.F1.F2, mail=Received, letter=Backward
time=222322294, nr=109, action=Initiated, from=F0.F1.F1, to=F0.F1, mail=Received, letter=Backward
time=222373648, nr=110, action=Accepted, from=F0.F1.F1, to=F0.F1, mail=Received, letter=Backward
time=222392699, nr=111, action=Completed, from=F0.F1.F1.F1, to=F0.F1.F1, mail=Processed, letter=Backward
time=222467108, nr=112, action=Accepted, from=F0.F1, to=F0, mail=Received, letter=Backward
time=222500797, nr=114, action=Completed, from=F0.F1.F1, to=F0.F1, mail=Processed, letter=Backward
time=222519349, nr=113, action=Accepted, from=F0.F2.F1, to=F0.F2, mail=Received, letter=Backward
time=222529922, nr=115, action=Initiated, from=F0.F1, to=F0, mail=Received, letter=Backward
time=222548805, nr=116, action=Completed, from=F0.F2.F1.F1, to=F0.F2.F1, mail=Processed, letter=Backward
time=222553588, nr=117, action=Initiated, from=F0.F1.F2, to=F0.F1, mail=Received, letter=Backward
time=222590381, nr=118, action=Accepted, from=F0.F2, to=F0, mail=Received, letter=Backward
time=222614551, nr=119, action=Completed, from=F0.F2.F2, to=F0.F2, mail=Processed, letter=Backward
time=222638302, nr=120, action=Initiated, from=F0.F2.F1, to=F0.F2, mail=Received, letter=Backward
time=222659792, nr=121, action=Completed, from=F0.F1, to=F0, mail=Processed, letter=Backward
time=222716104, nr=122, action=Initiated, from=F0.F2, to=F0, mail=Received, letter=Backward
time=222789666, nr=123, action=Accepted, from=F0.F2, to=F0, mail=Received, letter=Backward
time=222824920, nr=124, action=Completed, from=F0.F2.F1, to=F0.F2, mail=Processed, letter=Backward
time=222851296, nr=125, action=Accepted, from=F0.F1.F2, to=F0.F1, mail=Received, letter=Backward
time=222872791, nr=126, action=Initiated, from=F0.F2.F2, to=F0.F2, mail=Received, letter=Backward
time=222887923, nr=127, action=Completed, from=F0.F1.F2.F2, to=F0.F1.F2, mail=Processed, letter=Backward
time=222907896, nr=128, action=Completed, from=F0.F2, to=F0, mail=Processed, letter=Backward
time=222968726, nr=129, action=Accepted, from=F0.F1, to=F0, mail=Received, letter=Backward
time=222991622, nr=130, action=Completed, from=F0.F1.F2, to=F0.F1, mail=Processed, letter=Backward
time=223017268, nr=131, action=Initiated, from=F0.F2, to=F0, mail=Received, letter=Backward
time=223044061, nr=132, action=Initiated, from=F0.F1.F1, to=F0.F1, mail=Received, letter=Backward
time=223058595, nr=133, action=Accepted, from=F0.F2, to=F0, mail=Received, letter=Backward
time=223085724, nr=134, action=Completed, from=F0.F2.F2, to=F0.F2, mail=Processed, letter=Backward
time=223102743, nr=135, action=Initiated, from=F0.F2.F1, to=F0.F2, mail=Received, letter=Backward
time=223115188, nr=136, action=Completed, from=F0.F2, to=F0, mail=Processed, letter=Backward
time=223132295, nr=137, action=Initiated, from=F0.F1, to=F0, mail=Received, letter=Backward
time=223172194, nr=138, action=Accepted, from=F0.F2, to=F0, mail=Received, letter=Backward
time=223198866, nr=139, action=Completed, from=F0.F2.F1, to=F0.F2, mail=Processed, letter=Backward
time=223243176, nr=140, action=Accepted, from=F0.F1, to=F0, mail=Received, letter=Backward
time=223264569, nr=141, action=Completed, from=F0.F1.F1, to=F0.F1, mail=Processed, letter=Backward
time=223281755, nr=142, action=Initiated, from=F0.F1.F2, to=F0.F1, mail=Received, letter=Backward
time=223319350, nr=143, action=Completed, from=F0.F1, to=F0, mail=Processed, letter=Backward
time=223392468, nr=144, action=Accepted, from=F0.F1, to=F0, mail=Received, letter=Backward
time=223411071, nr=145, action=Completed, from=F0.F1.F2, to=F0.F1, mail=Processed, letter=Backward
time=223418991, nr=146, action=Initiated, from=F0.F2, to=F0, mail=Received, letter=Backward
time=223491459, nr=147, action=Completed, from=F0.F2, to=F0, mail=Processed, letter=Backward
time=223510512, nr=148, action=Initiated, from=F0.F2, to=F0, mail=Received, letter=Backward
time=223550030, nr=149, action=Completed, from=F0.F2, to=F0, mail=Processed, letter=Backward
time=223565760, nr=150, action=Initiated, from=F0.F1, to=F0, mail=Received, letter=Backward
time=223601013, nr=151, action=Completed, from=F0.F1, to=F0, mail=Processed, letter=Backward
time=223615946, nr=152, action=Initiated, from=F0.F1, to=F0, mail=Received, letter=Backward
time=223653427, nr=153, action=Completed, from=F0.F1, to=F0, mail=Processed, letter=Backward
As you see, this gets large very quickly. BTW, from=:x means the letter originates from an anonymous sender.
