[tf.data] Support eager mode benchmarking

[kvignesh1420]

This PR adds support for eager execution based benchmarks based on the benchmarking context. This is achieved by checking for the eager_execution() value within the benchmarking context and iterating over the dataset eagerly in the DatasetBenchmarkBase class.

Additionally, the ListFilesBenchmark in python/data/benchmarks/list_files_benchmark.py has been refactored to use the DatasetBenchmarkBase class.

A sample comparison w.r.t RangeBenchmark is as follows:

GRAPH MODE:

entry {
  name: "RangeBenchmark.modeling_on.graph"
  iters: 5
  wall_time: 1.7887322902679443e-07
  extras {
    key: "num_elements"
    value {
      double_value: 10000000.0
    }
  }
}

entry {
  name: "RangeBenchmark.modeling_off.graph"
  iters: 5
  wall_time: 1.541191530227661e-07
  extras {
    key: "num_elements"
    value {
      double_value: 50000000.0
    }
  }
}

EAGER MODE:

entry {
  name: "RangeBenchmark.modeling_on.eager"
  iters: 5
  wall_time: 2.0390429496765137e-07
  extras {
    key: "num_elements"
    value {
      double_value: 10000000.0
    }
  }
}

entry {
  name: "RangeBenchmark.modeling_off.eager"
  iters: 5
  wall_time: 1.8014323711395263e-07
  extras {
    key: "num_elements"
    value {
      double_value: 50000000.0
    }
  }
}

w.r.t ListFilesBenchmark

GRAPH MODE
entry {
  name: "ListFilesBenchmark.nested_directory(1024*16).graph"
  iters: 3
  wall_time: 8.092029020190239e-07
  extras {
    key: "num_elements"
    value {
      double_value: 2048.0
    }
  }
}

EAGER MODE
entry {
  name: "ListFilesBenchmark.nested_directory(1024*16).eager"
  iters: 3
  wall_time: 7.745111361145973e-07
  extras {
    key: "num_elements"
    value {
      double_value: 2048.0
    }
  }
}

NOTE: Description has been updated as per review comments.

[kvignesh1420]

cc: @aaudiber for review.

[aaudiber]

I don't think eager benchmarks are worth maintaining, since the backend dataset implementation works the same in both eager and graph mode. WDYT @jsimsa?

[jsimsa]

I think there is value in being able to run benchmarks in eager mode but this support should certainly not result in duplicating benchmark code. Instead, the benchmark base should be modified to make it possible to run a benchmark both in graph mode or eagerly. Which mode to use could be a based on which parameter is passed to python/data/benchmarks/benchmark_base.py run_benchmark method, defaulting to the mode of the context in which the benchmark is executed.

[kvignesh1420]

I see that not all the benchmarks located in python/data/benchmarks/ and python/data/experimental/benchmarks/ subclass the DatasetBenchmarkBase class. Thus, making changes to the DatasetBenchmarkBase class will cover a few of them but not all. This is due to the custom _benchmark logic that is needed instead of the default one available in DatasetBenchmarkBase.

As per @jsimsa 's suggestion, I can modify the run_benchmark method of DatasetBenchmarkBase class to take a parameter named eager which will either be True or False. It will default to False so that it doesn't disturb the current benchmark scenarios. I can then add conditional logic to handle eager=True case within that method. The benchmarks which inherit the DatasetBenchmarkBase can now call the run_and_report_benchmark method with the option of benchmarking eagerly.

I can do the same for benchmarks which do not inherit the DatasetBenchmarkBase class and modify the _benchmark method to take the eager parameter and conditionalize the logic.

What do you think? @jsimsa @aaudiber?

UPDATE: The benchmarks do not take the eager parameter but execute in either eager or graph mode based on context.

[jsimsa]

Thanks @kvignesh1420. You plan for modifying DatasetBenchmarkBase sounds good to me. It would also be good to understand why do some benchmarks need custom _benchmark logic and whether it would be possible to refactor things so that all benchmarks inherit from DatasetBenchmarkBase.

[kvignesh1420]

@jsimsa thanks for the go-ahead. I will update the PR with the required changes and try to refactor things.

[kvignesh1420]

I have modified the DatasetBenchmarkBase class to support eager iteration over the datasets depending on the context. Additionally, the ListFilesBenchmark in python/data/benchmarks/list_files_benchmark.py has been refactored to use the DatasetBenchmarkBase class. The refactor changes the current output of the benchmark in order to be in sync with the style of other benchmarks.

cc: @jsimsa @aaudiber let me know how it looks. I can take up the benchmarks in data/experimental/benchmarks after this.

[jsimsa]

Thank you. As per one of my comments, we want to test that the new implementation works as expected before merging the PR (and I provided you with instructions on how to do so).

[kvignesh1420]

@jsimsa I have modified the run_and_report_benchmark method in DatasetBenchmarkBase to append the mode in which the benchmarks are running to the benchmark name. Also, a sample benchmark result has been added to the description. If you want additional results from eager mode benchmarks and its graph counter-part, I can attach the files with the log outputs. Let me know.

[jsimsa]

This looks great. Thank you for the changes. Left one minor comment. Other than that it looks good.