usage: memory_tracer.py [-h]
                        (--workload_path_on_guest WORKLOAD_PATH_ON_GUEST | --workload_path_on_host WORKLOAD_PATH_ON_HOST)
                        (--analysis_tool_path ANALYSIS_TOOL_PATH | --trace_fifo_path TRACE_FIFO_PATH | --dont_trace | --dont_use_qemu)
                        [--trace_only_CPL3_code_GMBE]
                        [--log_of_GMBE_block_len LOG_OF_GMBE_BLOCK_LEN]
                        [--log_of_GMBE_tracing_ratio LOG_OF_GMBE_TRACING_RATIO]
                        [--timeout TIMEOUT | --dont_add_communications_with_host_to_workload]
                        [--print_trace_info] [--dont_use_nographic]
                        [--dont_exit_qemu_when_done]
                        [--guest_RAM_in_MBs GUEST_RAM_IN_MBS] [--verbose]
                        guest_image_path snapshot_name qemu_with_GMBEOO_path

Run a workload on a single-core QEMU guest while writing optimized GMBE trace records to a FIFO or to an analysis tool.
(See https://csl-wiki.cs.technion.ac.il/mediawiki/index.php/Qemu_tracing for more high level info. Don't worry if you don't have permissions. There isn't any essential info there.)

(memory_tracer.py assumes you have already run build.py successfully. See SETUP_README for setup instructions.)

GMBE is short for guest_mem_before_exec. This is an event in upstream QEMU 3.0.0 that occurs on every attempt of the QEMU guest to access a virtual memory address. (For more about tracing in upstream QEMU, see qemu/docs/devel/tracing.txt in upstream QEMU's sources.)

We optimized QEMU's tracing code for the case in which only trace records of GMBE are gathered. (We call it GMBE only optimization - GMBEOO, and so we gave our fork of QEMU the name qemu_with_GMBEOO. Note that in our documentation and comments, we often refer to qemu_with_GMBEOO as `qemu`.)
When GMBEOO is enabled (in qemu_with_GMBEOO), a trace record is structured as follows:

#pragma pack(push, 1) // exact fit - no padding
struct GMBEOO_TraceRecord {
    uint8_t     size_shift  : 3; /* interpreted as "1 << size_shift" bytes */
    bool        sign_extend : 1; /* whether it is a sign-extended operation */
    uint8_t     endianness  : 1; /* 0: little, 1: big */
    bool        store       : 1; /* whether it is a store operation */
    uint8_t     cpl         : 2; /* probably the CPL while the access was performed.
                                    "probably" because we consistently see few trace
                                    records according to which CPL3 code tries to
                                    access cpu_entry_area, which shouldn't be
                                    accessible by CPL3 code. For more, see
                                    https://unix.stackexchange.com/questions/476768/what-is-cpu-entry-area,
                                    including the comments to the answer. */
    uint64_t    unused2     : 55;
    bool        is_valid    : 1;  /* whether the trace record is ready to be written
                                     to the trace file. This field is for internal
                                     use by qemu_with_GMBEOO, and is useless for the
                                     analysis tool, as it would always be 1. */
    uint64_t    virt_addr   : 64; /* the virtual address */
};
#pragma pack(pop) // back to whatever the previous packing mode was

memory_tracer.py also prints the workload info (in case it isn't the empty string), and the tracing duration in milliseconds.
(See the documentation of --dont_add_communications_with_host_to_workload below for how to use "workload info".)In case --analysis_tool_path is specified, memory_tracer.py also prints the output of the analysis tool.

Note that the given workload can be any executable (e.g. ELF, bash script).

If --analysis_tool_path is specified, the provided analysis tool must do the following:
1. Receive in argv[1] the path of the trace FIFO, but not open it for reading yet.
2. Receive in argv[2:] the workload info. (This isn't a requirement, but an optional feature.)
3. Register a handler for the signal SIGUSR1 (e.g. by calling the `signal` syscall). The handler must:
    a. Print "-----begin analysis output-----".
    b. Print the output of the analysis tool.
    c. Print "-----end analysis output-----".
4. Print "Ready to analyze" when you wish the tracing to start.
5. Open the trace FIFO for read, and start reading trace records from it. Note that the reading from the FIFO should be as fast as possible. Otherwise, the FIFO's buffer would get full, and qemu_with_GMBEOO would start blocking when it tries to write to the FIFO. Soon, trace_buf (the internal trace records buffer in qemu_with_GMBEOO) would get full, and trace records of new GMBE events would be dropped.
(If any of the messages isn't printed, it will probably seem like memory_tracer.py is stuck.)

Note that some of the command line arguments might be irrelevant to you as a user of memory_tracer, but they exist because they are useful while developing memory_tracer.

simple usage examples: 
(1)
~/qemu_mem_tracer/memory_tracer.py ~/oren_vm_disk2.qcow2 ready_for_memory_tracer ~/qemu_with_GMBEOO --analysis_tool_path ~/qemu_mem_tracer/tests/toy_workloads_and_analysis_tools/tests_bin/simple_analysis --workload_path_on_host /bin/date
-----> memory_tracer does the following: Start a qemu guest (that was specially prepared for memory_tracer) using the disk image ~/oren_vm_disk2.qcow2 and the internal snapshot `ready_for_memory_tracer`; Send the workload /bin/date from the host to the guest; Run the analysis tool ~/qemu_mem_tracer/tests/toy_workloads_and_analysis_tools/tests_bin/simple_analysis on the host and run the workload (that was sent from the host) on the guest, while sending trace records to the analysis tool.

(2)
~/qemu_mem_tracer/memory_tracer.py ~/oren_vm_disk2.qcow2 ready_for_memory_tracer ~/qemu_with_GMBEOO --analysis_tool_path ~/qemu_mem_tracer/tests/toy_workloads_and_analysis_tools/tests_bin/simple_analysis --workload_path_on_guest /bin/date
-----> Same as (1), but with a workload that was already inside the guest.

(3)
mkfifo ~/tmp_example_fifo
cat ~/tmp_example_fifo > ~/trace_records.bin &
~/qemu_mem_tracer/memory_tracer.py ~/oren_vm_disk2.qcow2 ready_for_memory_tracer ~/qemu_with_GMBEOO --trace_fifo_path ~/tmp_example_fifo --workload_path_on_guest /bin/date
rm ~/tmp_example_fifo
-----> memory_tracer starts a qemu guest using the disk image ~/oren_vm_disk2.qcow2 and the internal snapshot `ready_for_memory_tracer`, and runs the workload /bin/date (that was already inside the guest) on the guest, while sending trace records to the FIFO ~/tmp_example_fifo.

positional arguments:
  guest_image_path      The path of the qcow2 file which is the image of the
                        guest.
  snapshot_name         The name of the snapshot saved by the monitor command
                        `savevm`, which was specially constructed for
                        memory_tracer, according to SETUP_README.
  qemu_with_GMBEOO_path
                        The path of qemu_with_GMBEOO.

optional arguments:
  -h, --help            show this help message and exit
  --workload_path_on_guest WORKLOAD_PATH_ON_GUEST
                        The path of the workload on the guest.
  --workload_path_on_host WORKLOAD_PATH_ON_HOST
                        The path of the workload on the host. The file in that
                        path would be sent to the guest to run as the
                        workload. In other words, if this file isn't quite
                        small, you would be better off copying it into the
                        qemu guest (e.g. by using scp) and saving a snapshot,
                        and then using --workload_path_on_guest.
  --analysis_tool_path ANALYSIS_TOOL_PATH
                        Path of an analysis tool that would start executing
                        before the tracing starts. See more requirements for
                        it in the general description of memory_tracer.py
                        above.
  --trace_fifo_path TRACE_FIFO_PATH
                        Path of the FIFO into which trace records will be
                        written. Note that as mentioned above, a scenario in
                        which the FIFO's buffer getting full is bad, and so it
                        is recommended to use a FIFO whose buffer is of size
                        `cat /proc/sys/fs/pipe-max-size`.
  --dont_trace          If specified, memory_tracer.py will run without
                        enabling the tracing feature of qemu_with_GMBEOO.
                        Therefore, it will not print the trace info (even if
                        --print_trace_info is specified). This is useful for
                        comparing the speed of qemu_with_GMBEOO with and
                        without tracing. Note that code that does such a
                        comparison has already been implemented in
                        tests/tests.py. See the function
                        `_test_toy_workload_duration_and_MAPS`.
  --dont_use_qemu       If specified, memory_tracer.py will run the workload
                        on the host (i.e. natively). Please pass dummy non-
                        empty strings as the arguments guest_image_path,
                        snapshot_name, and qemu_with_GMBEOO_path. As expected,
                        no trace info will be printed (even if
                        --print_trace_info is specified).This is useful for
                        comparing the speed of qemu_with_GMBEOO to running the
                        code natively. Note that code that does such a
                        comparison has already been implemented in
                        tests/tests.py. See the function
                        `_test_toy_workload_duration_and_MAPS`.
  --trace_only_CPL3_code_GMBE
                        If specified, qemu would only trace memory accesses by
                        CPL3 code. Otherwise, qemu would trace all accesses.
  --log_of_GMBE_block_len LOG_OF_GMBE_BLOCK_LEN
                        Log of the length of a GMBE_block, i.e. the number of
                        GMBE events in a GMBE_block. (It is used when
                        determining whether to trace a GMBE event).
                        For example, if log_of_GMBE_block_len is 4, then the
                        GMBE_block size is 2^4=16. If there were 64 memory
                        accesses, then the tracer will have 4 blocks
                        (GMBE_blocks) of memory accesses to trace, each block
                        for 16 memory accesses.
  --log_of_GMBE_tracing_ratio LOG_OF_GMBE_TRACING_RATIO
                        Log of the ratio between the number of blocks of GMBE
                        events we trace to the total number of blocks. E.g. if
                        log_of_GMBE_tracing_ratio is 4, then
                        GMBE_tracing_ratio is 16. Thus, we trace 1 block, then
                        discard 15 blocks, then trace 1, then discard 15, and
                        so on. The default is 0, which means that the default
                        tracing ratio is 1, i.e. all of the blocks are traced.
  --timeout TIMEOUT     If specified, the workload would be stopped when the
                        specified timeout (in seconds) elapses.
  --dont_add_communications_with_host_to_workload
                        If specified, the workload would not be wrapped with
                        code that handles the required communications between
                        the guest and the host, i.e. the workload (given in
                        workload_path_on_host or workload_path_on_guest) must
                        do the following: (1) Print "-----begin workload
                        info-----". (2) Print runtime info of the workload.
                        This info will be written to the stdout of
                        memory_tracer, as well as passed as cmd arguments to
                        the analysis tool in case --analysis_tool_path was
                        specified. (Print nothing if you don't need any
                        runtime info.) (3) Print "-----end workload
                        info-----". (4) Print "Ready to trace. Press enter to
                        continue" when you wish the tracing to start. (5) Wait
                        until enter is pressed, and only then start executing
                        the code you wish to run while tracing. (`getchar();`
                        in C or `read -n1` in bash would do.) (6) Print "Stop
                        tracing" when you wish the tracing to stop. (If any of
                        these messages isn't printed, it will probably seem
                        like memory_tracer.py is stuck.)
  --print_trace_info    If specified, memory_tracer.py would also print some
                        additional trace info:
                        num_of_events_waiting_in_trace_buf (only if it isn't
                        0, which probably shouldn't happen);
                        num_of_GMBE_events_since_enabling_GMBEOO (excluding
                        non-CPL3 GMBE events, in case
                        --trace_only_CPL3_code_GMBE was specified);
                        num_of_events_written_to_trace_buf;
                        num_of_missing_events (i.e.
                        `num_of_events_written_to_trace_buf -
                        num_of_events_written_to_trace_file -
                        num_of_events_waiting_in_trace_buf`, but only if it
                        isn't 0, which is probably a bug in qemu_with_GMBEOO);
                        actual_tracing_ratio (i.e.
                        num_of_GMBE_events_since_enabling_GMBEOO /
                        num_of_events_written_to_trace_buf);
                        num_of_dropped_events (only if it isn't 0 - this is
                        the number of events such that when qemu_with_GMBEOO
                        tried to write them to the trace_buf (its internal
                        trace records buffer), it was full, so they were
                        discarded, which shouldn't happen normally.
  --dont_use_nographic  If specified, qemu_with_GMBEOO will be started with
                        the cmd argument `-monitor stdio` instead of
                        `-nographic`. This degrades performance, but is
                        probably more convenient while developing
                        memory_tracer on your machine. See the official
                        documentation (https://qemu.weilnetz.de/doc/qemu-
                        doc.html) for more about `-nographic` and `-monitor
                        stdio`.(Note that this option probably makes no sense
                        in case you are running memory_tracer.py on a remote
                        server using ssh.)
  --dont_exit_qemu_when_done
                        If specified, qemu won't be terminated after running
                        the workload, and you would be able to use the
                        terminal to send monitor commands, as well as use the
                        qemu guest directly, in case --dont_use_nographic was
                        specified.Anyway, you would be able to use the qemu
                        guest, e.g. by connecting to it using ssh.
  --guest_RAM_in_MBs GUEST_RAM_IN_MBS
                        The startup RAM size (in mega-bytes) of the qemu
                        guest. This is simply passed to qemu_with_GMBEOO as
                        the -m argument. See the official documentation
                        (https://qemu.weilnetz.de/doc/qemu-doc.html). The
                        default is 2560. Note that qemu would terminate
                        immediately if you specify a different RAM size than
                        the one that was specified when the internal snapshot
                        was created.
  --verbose, -v         If specified, memory_tracer's debug messages are
                        printed.




future work (i.e. ideas for ways to improve memory_tracer):
1. Squeeze `GMBEOO_TraceRecord` into 8 bytes (the current implementation of
   qemu_with_GMBEOO assumes that `sizeof(GMBEOO_TraceRecord)` is a power of 2,
   and it is currently 16 bytes). To achieve this, you can use some of the
   most significant bits in a 64bit virtual address (as they are always equal to the
   most significant bit (see
   https://www.kernel.org/doc/Documentation/x86/x86_64/mm.txt)) to store
   the CPL of the code that made the memory access, as well as whether it was
   a load/store, etc.
2. With regard to optimizing the code that handles a GMBE event, it might be
   better to not extract the CPL in the handler of a GMBE event, and instead
   extract it in the handler of an event that occurs on every CPL change (and
   just store the current CPL in a global variable).
3. Lluis Vilanova (who is also a PostDoc in the Technion) has done a lot of
   work on qemu. Some of his work wasn't merged into upstream qemu, and he said
   that we might leverage this work to significantly improve the performance of
   memory_tracer (though it would require a non-trivial amount of work). Lluis'
   work on qemu can be found (if I understand correctly) at
   https://projects.gso.ac.upc.edu/projects/qemu-dbi.
4. qemu_with_GMBEOO was forked from qemu tag v3.0.0 (August 2018). I guess
   the performance of upstream qemu would improve in the future, so merging
   upstream into qemu_with_GMBEOO would probably help.
   Note that merging upstream qemu into qemu_with_GMBEOO might break
   `GMBEOO_add_cpl_to_GMBE_info_if_should_trace` (it is quite obvious from the
   code of this short function what might break it, and how to easily fix that).
5. With regard to optimizing `writeout_thread` (in qemu_with_GMBEOO/trace/simple.c),
   you might use a bitmap of `TRACE_BUF_LEN / sizeof(GMBEOO_TraceRecord)` bits
   to store for each trace record in trace_buf whether it is valid or not. Then,
   you might also use x86's bit scan instructions to find the first invalid
   trace record.
6. Make memory_tracer support also a multi-core guest. I think that the main
   thing you would have to do is to think whether our code in
   qemu_with_GMBEOO/trace/simple.c is thread safe (and change it if it isn't).
   You would probably also want to add an identifier of the CPU that performed
   the memory access to `struct GMBEOO_TraceRecord`.