Regain performance by caching initializer names in ORTModule

[baijumeswani]

Every call to self._graph_info.initializer_names or self._graph_info.initializer_names_to_train resulted in a O(n) lookup time. Although these two were Python sets, their lookup time was linear. This was happening most likely because on every reference, pybind made a fresh copy of the C++ unordered_set to a Python set.

While we research ways to optimize this, this pull request fixes the perf regression that #7631 introduced by caching the set of initalizer names on the frontend.

This pull request also adds more unit tests for testing the support for unused model parameters.

[thiagocrepaldi]

LGTM
I am assuming you ran Ravi's script which benchmarks all scenarios

[baijumeswani]

LGTM
I am assuming you ran Ravi's script which benchmarks all scenarios

I ran Ravi's script for one of the models and it looked good. I am assuming this will generalize to other models as well.

[mrry]

Thanks for figuring out the problem Baiju! I would not have expected that property access to be causing a set construction every time :).

We should definitely merge this as-is, but one possibility for fixing the TODO is to investigate the trick @codemzs used to avoid copying STL structures (vectors in this case) across the Python/C++ boundary:

onnxruntime/orttraining/orttraining/python/orttraining_pybind_state.cc

Line 19 in 7bb3f24

PYBIND11_MAKE_OPAQUE(std::vector<OrtValue>);

[ytaous]

do u know what's perf gain by % with Ravi's?

---

In reply to: 840811250

[baijumeswani]

@ytaous no additional gain. Just back to where we were before pull request #7631 caused the regression.