Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce abstractions for loggable events #75

Closed
kuhar opened this issue Sep 28, 2022 · 2 comments
Closed

Introduce abstractions for loggable events #75

kuhar opened this issue Sep 28, 2022 · 2 comments
Assignees

Comments

@kuhar
Copy link
Contributor

kuhar commented Sep 28, 2022

The existing event logging is rather ad-hoc. Examples:

  FrameTimeLayerData(char* log_filename, uint64_t exit_frame_num_or_invalid,
                     const char* benchmark_watch_filename,
                     const char* benchmark_start_string)
      : LayerData(log_filename, "Frame Time (ns),Benchmark State"),
        exit_frame_num_or_invalid_(exit_frame_num_or_invalid),
        benchmark_start_pattern_(StrOrEmpty(benchmark_start_string)) {
    LogEventOnly("frame_time_layer_init");
  layer_data->LogTimeDelta("frame_present",
                           layer_data->HasBenchmarkStarted() ? "1" : "0");
    int64_t logged_delta = ToInt64Nanoseconds(now - last_log_time_);
    LogLine(event_type, CsvCat(logged_delta, extra_content), now);
    layer_data->LogEventOnly("frame_time_layer_exit",
                             absl::StrCat("terminated,frame:", frames_elapsed));

This form of logging is unstructured and does not allow for emitting logs in different output formats. All the event data is converted to strings and concatenated before we can decide which logging format to use.

This task is to come up with a more structured abstraction for logged events that will allow for multiple output formats (e.g., csv and json).

The generic interface should provide the following functions:

  1. getEventName()
  2. getLogLevel() -- decides the 'verbosity level', e.g., whether to log to CSV or to the common log file
  3. getNumAttributes -- returns the number of data entries (or 'columns' / tuple elements / etc)
  4. getAttributes()

We want to support a few data entry types:

  1. String
  2. Int64 -- e.g., count of something
  3. Int64Array -- e.g., array of hashes
  4. Timepoint / timestamp -- when something happened
  5. Duration -- how long something took

All of this combines as follows:

  • Event is a names set of attributes
  • Attribute is a key-value pair where the attribute name is the key and value can be of one of the supported types.

A concrete layer-specific event definition could look something like this:

class ShaderModuleCreateEvent : public LoggableEvent {
public:
   // (Constructor and other functions...)
   
  std::vector<const Attribute*> getAttributes() const {
    return {&shader_module_hashes_, &creation_time_};
  }
  
private:
  Int64ArrayAttr shader_module_hashes_("shader_hashes");
  DurationAttr creation_time_("creation_time");
};

std::unique_ptr<LoggableEvent> CreateShaderModuleCreateEvent(const ShaderHashes &hashes,
                                                             Duration creation_time,
                                                             Timestamp when = Timestamp::Now()) {
  return std::make_unique<ShaderModuleCreateEvent>("create_shader_module", when,
                                                   hashes, creation_time);
} 

(This is just an example for inspiration. Better ideas welcome!)

@kuhar kuhar mentioned this issue Sep 28, 2022
7 tasks
@kuhar kuhar assigned kuhar and miladHakimi and unassigned kuhar Sep 28, 2022
@kuhar
Copy link
Contributor Author

kuhar commented Sep 28, 2022

A possible attribute implementation: https://godbolt.org/z/4j4bMr1bc

kuhar pushed a commit that referenced this issue Oct 11, 2022
Currently, the logging mechanism is unstructured. Various log functions
with different interfaces are used to generate the logs. All these
functions receive a series of inputs, combine them in a string with csv
format, and write them in the standard output or a file.
`The`Event` class unifies the interface for creating logs. It provides
these two functionalities:
1) Using the same interface for creating all the logs to replace the
current ad-hoc event logging functions.
2) Allowing log generation for different formats other than csv.

Each `Event` has a name and a set of attributes. Attributes contain the
value and its name. The name is a string telling what the attribute is
(e.g, duration, timestamp, etc).

The current log functions use only a limited set of types; string,
int64, and vector. Hence, right now, `Attribute` has three
implementations covering the mentioned types.

This PR also contains the implementation for Event, an event-to-csv
generator, and simple unit tests to check if they work as expected.

Issue: #75
@kuhar
Copy link
Contributor Author

kuhar commented Nov 5, 2022

I consider this completed. @miladHakimi updated all layers to use the new abstractions for CSV logging

@kuhar kuhar closed this as completed Nov 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants