The right way to re-parse with a different converge time?

[higher-order]

I have run tornettools parse on a few directories with finished simulation and plotted them. Now I want to parse them with a different converge time. Is -s the right parameter to use?

tornettools parse -s -c 16200 <tornet dir>

If I then run tornettools plot, will it follow the new converge time?

[robgjansen]

You should not use the -s ie --skip-raw option if you are also going to use the --converge-time option. The reason is that if you change the converge time, the parse script will filter the download times from the tgen log files differently (it will ignore downloads that occur during the converge window). If you use the -s flag, you will skip re-parsing the tgen log files and thus that filtering step will not be re-run and the tgen stats will be stale (parsed using the original converge time).

Note that tornettools parse --help provides some description of the options. Here are the relevant ones for this discussion:

  -s, --skip-raw        Skip trying to parse raw logs from <tornet_config_path>/shadow.data/hosts into <tornet_config_path>/*.json.xz,
                        continuing on to parse <tornet_config_path>/*.json.xz into <tornet_config_path>/tornet.plot.data/*.json. This is
                        useful if you no longer have those raw logs, but want to re-parse and/or re-plot with updated post-processing.
                        (default: False)
  -c N, --converge-time N
                        The number of seconds after the beginning of the simulation that we should ignore in the tgen and oniontrace log
                        files, i.e., so we don't track network performance before the network has reached steady-state. Log messages during
                        the interval [0, converge_time) will be ignored. (default: 1200)
  -r N, --run-time N    The number of seconds after the converge-time is reached that we should consider the tgen and oniontrace log files
                        valid for the purpose of extracting performance stats. Log messages during the interval [converge_time, converge_time
                        + run_time) will be parsed. A negative value indicates that we should parse all available data, i.e, during the
                        interval [converge_time, ∞). (default: -1)

[higher-order]

@robgjansen Thank you for the explanation. So the simulation is 5-hour long. I first parsed it with -c 7200 and plotted. I then parsed it again with -c 16200 and plotted. But the two plots look visually identical to each other. That's when I tried parsing with -s -c 16200. The plot became different, but from your explanation, it sounds like it was using the tgen stats from -c 7200?

But if that's the case, why would the plot seem different with -s, but identical without it? More importantly, what is the correct way to parse the same simulation again with a different converge time?

[robgjansen]

The feature was added in #45, in commit 2555f9f.

Here is the latest code:

tornettools/tornettools/parse.py

Lines 13 to 32 in df6ada5

	logging.info("Parsing tgen logs.")
	if args.skip_raw or parse_tgen_logs(args):
	logging.info("Extracting tgen plot data.")
	extract_tgen_plot_data(args)
	else:
	logging.warning("Parsing tgen logs failed, so we cannot extract tgen plot data.")
	logging.info("Parsing oniontrace logs.")
	if args.skip_raw or parse_oniontrace_logs(args):
	logging.info("Extracting oniontrace plot data.")
	extract_oniontrace_plot_data(args)
	else:
	logging.warning("Parsing oniontrace logs failed, so we cannot extract oniontrace plot data.")
	logging.info("Parsing resource usage logs.")
	if args.skip_raw or parse_resource_usage_logs(args):
	logging.info("Extracting resource usage plot data.")
	extract_resource_usage_plot_data(args)
	else:
	logging.warning("Parsing resource usage logs failed, so we cannot extract resource usage plot data.")

As you can see, the follow three functions will only be called if --skip-raw is False (the default). If you set --skip-raw to True, the second part of the or clause (i.e., the functions below that re-parse the log files) will be skipped.

• parse_tgen_logs(args)
• parse_oniontrace_logs(args)
• parse_resource_usage_logs(args)

Since it seems that you do want these functions to run with the new converge timeframe, the correct way to run it as far as I can tell would be:

tornettools parse --converge-time 16200 <tornet dir>

But the behavior you describe sure sounds like it is doing the opposite, so I'm not sure what is going on in your case. If you are unsure what is going on, I suggest that you delete all of the json.xz files in tornetdir/* and try again from scratch.

[robgjansen]

You might also want to make sure that there is actually data to be parse in the timeframe [16200, 18000), otherwise when it re-parses the logs with that converge time and finds empty data, I'm not sure if it will rewrite new (empty) json files.

[higher-order]

@robgjansen Turns out it wasn't a problem with tornettools at all. The latest Ubuntu upgrade broke most of the Python packages on the system. And tgentools and oniontracetools were not available. The parse command showed the errors but I didn't notice them. I only realized the problem after I tried deleting the parsing artefacts before re-parsing, and seeing that many lines are missing. I re-installed tgentools and oniontracetools and it should be working now.

It may be helpful if the tgentools and oniontools errors are somehow highlighted in the output of the parse command.