API usage logging within TorchVision

[kazhang]

Goal

To understand TorchVision usage within an organization(e.g. Meta).

The events give insights into torchvision usage with regards to individual callsites, workflows etc. The organization could also learn the trending APIs, which could be used to guide component development/deprecation etc.

Policy

• Usage should be recorded only once for the same API within a process;
• We should record events as broadly as possible, duplicated events(e.g. module and function log the same thing) is OK and can be dedup in downstream pipelines.
• For modules, API usage should be recorded at the beginning of constructor of the main class. For example __init__ of RegNet, but not on ones of submodules(e.g. ResBottleneckBlock)
• For functions, API usage should be recorded at the beginning of the method;
• For torchvision.io, the logging must be added both on the Python and the C++ (using the csrc submodule as mentioned) side.
• On torchvision.ops, the calls should be added both on the main class of the operator (eg StochasticDepth) and on its functional equivalent (eg stochastic_depth) if available.
• On torchvision.transforms, the calls should be placed on the constructors of the Transform classes, the Auto-Augment classes and the functional methods.
• On torchvision.datasets, the calls are placed once on the constructor of VisionDataset so we don't need to add them individually on each dataset.
• On torchvision.utils, call should be added to the top of each public method.

Event Format

Full qualified name of the component is logged as the event. For example: torchvision.models.resnet.ResNet
Note: for events from C++ APIs, “.csrc” should be added after torchvision, for example: torchvision.csrc.ops.nms.nms

Usage Log API

• C++: C10_LOG_API_USAGE_ONCE()
• Python:

from ..utils import _log_api_usage_once
# for class
_log_api_usage_once(self)
# for method
if not torch.jit.is_scripting() and not torch.jit.is_tracing():
  _log_api_usage_once(nms) 

Above APIs are lightweight. By default, they are just no-op. It’s guaranteed that the same event is only recorded once within a process. Please note that 8 GPUs will still lead to 8 events, so the events should be dedup by a unique identifier like workflow job_id.

Implementation

def _log_api_usage_once(obj: Any) -> None:
    if not obj.__module__.startswith("torchvision"):
        return
    name = obj.__class__.__name__
    if isinstance(obj, FunctionType):
        name = obj.__name__
    torch._C._log_api_usage_once(f"{obj.__module__}.{name}")

Also considered

• log usage in base class
  • Create a base class for all models, datasets, transforms and log usage in the init of base class
  • Introducing extra abstraction only for logging seems overkill. In Base class for models #4569, we couldn't find any other features to be added to model base class; In addition, we also need a way to log non-class usage;
• use decorator
  • For example: @log_api_usage in use decorator to log API usage #4976
  • doesn’t work with TorchScript since decorator needs to use kwargs, which is not supported in TorchScript;
• use function’s __module__:
  • For example: _log_api_usage(nms.__module__, "nms")
  • doesn’t work with TorchScript: attribute lookup is not defined on function
• use global constant for module
  • For example: _log_api_usage(MODULE, "nms")
  • doesn’t work with TorchScript;
• use flat namespace
  • For example: log events as "torchvision.{models|transforms|datasets}.{class or function name}"
  • there might be name collisions;
• use object or function as param in logging API
  • For example: _log_api_usage_once(self)
  • doesn’t work with TorchScript;
• log fully qualified name with __qualname__ for class, string for function
  • For example: use fully qualified name #5096
  • too cumbersome for functions

[datumbox]

Thanks for the write up @kazhang.

I agree with the direction of the proposal. Personally I would favour torchvision.{ models | transforms | ops | io}.{function | class} but no strong opinions.

As discussed at #5038 (comment), adopting a straightforward, easy to implement and review solution will help us ensure that logging will be added on new endpoints and models. Already we've missed it in a couple of places (see #5044, admittedly I was involved on the reviews and missed this).

It might be good to include a few clarifications on how the proposal is implemented in practice, so that open-source contributors have clearer understanding of the policy. For example:

1. On torchvision.models, the logging call should be added on the constructor of the main class (eg RegNet) not on the ones of submodules (eg ResBottleneckBlock).
2. On torchvision.io, the logging must be added both on the Python and the C++ (using the csrc submodule as mentioned) side.
3. On torchvision.ops, the calls should be added both on the main class of the operator (eg StochasticDepth) and on its functional equivalent (eg stochastic_depth) if available. Note that this is not the situation as of now and this needs to be updated on the ops submodule.
4. On torchvision.transforms, the calls should be placed on the constructors of the Transform classes, the Auto-Augment classes and the functional methods.
5. On torchvision.datasets, the calls are placed once on the constructor of VisionDataset so we don't need to add them individually on each dataset.
6. On torchvision.utils, I assume we should put method calls on the top of each public method.

Concerning the outstanding work, my understanding is that:

• torchvision.models is done.
• torchvision.datasets is done.
• torchvision.io has an open PR at Log io usage #5038 which requires minor changes but mostly done
• torchvision.ops needs to be updated to include calls in all class constructors. I believe that the calls on all methods already exists.
• torchvision.transforms has no logging
• torchvision.utils has no logging

@kazhang would you be OK to open a ticket for ops, transforms and utils and provide a sample implementation for each on the issue? The open-source community can definitely help us tackle this one. :)

[kazhang]

In terms of format, I personally prefer torchvision.{ models | transforms | ops | io | util}.{function | class} too, we'd need to update the format string here though. In this case, I recommend to change to logging API to

def _log_api_usage_once(module:str, name: str) -> None:
    if torch.jit.is_scripting() or torch.jit.is_tracing():
        return
    torch._C._log_api_usage_once(f"torchvision.{module}.{name}")

So that the API is consistent across module and function(as of today, module uses self, function uses string.

[datumbox]

@kazhang After reviewing some of the PRs, I noticed that:

• On C++ (see pr Log io usage #5038) the torchvision/csrc/io/image/cpu/decode_jpeg.cpp records to the logger torchvision.csrc.io.decode_jpeg. Strictly speaking, this method doesn't really exist and might conflict with other methods within the io namespace (perhaps the gpu equivalents if on the future manage to use the dispatcher).
• On Python (see pr Add api usage log to transforms #5007) the torchvision.transforms.functional.get_image_size records to the logger torchvision.transforms.get_image_size. Again this doesn't really exist because the methods under functional are not imported directly to the transforms namespace.
• On Python (see pr revamp log api usage method #5072), the torchvision.models.detection.RetinaNet is recorded as torchvision.models.RetinaNet which again doesn't exist.
• On Python (see pr Add logging to torchvision ops #4799), the torchvision.ops.boxes.batched_nms method was recorded as torchvision.ops.batched_nms. Though the method actually exists in that location due to being imported on the ops namespace, it's not the fully qualified name.

First of all we should note that some of the inconsistency preexisted your PRs/proposal. Moreover your proposal to log each method under the flat namespace does simplify things and at first glance looks very appealing. It also avoids recording very long fully qualified names such as torchvision.csrc.io.image.cpu.decode_jpeg.decode_jpeg. Unfortunately it also opens up the possibility for name collisions and potentially misleading method locations. What are your thoughts on this?

cc @NicolasHug

[NicolasHug]

Regarding Python, I think we could try to record the up-most subpackage (with an __init__.py file) where the function is available. This reflects how we expect our users to import these functions / methods. By "upmost" I mean the one that's closest to the tochvision. namespace

For example we explicitly expose batched_nms in torchvision.ops.__init__.py, so it makes sense to log torchvision.ops.batched_nms.

Similarly, since RetinaNet is not available in torchvision.models.__init__.py, I would agree that torchvision.models.detection.RetinaNet makes more sense.

[datumbox]

For Python, recording the most compact valid name path of the method/class is reasonable. If that was the case, would you make _log_api_usage_once receive 2 params as is now or a single string as before? Both are possible aka:

_log_api_usage_once("models.detection", "RetinaNet")
# VS
_log_api_usage_once("torchvision.models.detection.RetinaNet")

Also how about the C++ side? Using the fully qualified name such as torchvision.csrc.io.image.cpu.decode_jpeg.decode_jpeg sounds like an overkill. An alternative might be to use a path similar to how the methods are registered (see for example decode_jpeg).

It's probably worth clarifying these details fully prior making more changes to the code to minimize throwaway work. Let me know your thoughts, thanks!

[NicolasHug]

use a path similar to how the methods are registered (see for example decode_jpeg).

Agree with this if we're sure to only ever log functions that we register through the dispatcher. Otherwise, we still need a convention for logging the non-registered functions, in which case I would go with this:

Using the fully qualified name such as torchvision.csrc.io.image.cpu.decode_jpeg.decode_jpeg sounds like an overkill

It's a bit cumbersome I agree. But it also seems to be the most straightforward, dumb simple, and name-clash free strategy.

We can do both BTW: use the first strategy for registered functions, and use the other one as a fall-back

[kazhang]

@NicolasHug @datumbox
Yes, I agree that the drawback of using flat namespace is name collision despite its simplicity.
With regards to the Python API, one consideration is model inheritance for modules, we can't simply use

_log_api_usage_once("models", "RetinaNet")

Because the model inherit from this class won't be tracked properly, instead, we should use

_log_api_usage_once("models", self.__class__.__name__)

Now, if we want to record the up-most subpackage, we can do

_log_api_usage_once("models.detection.", self.__class__.__name__)

But if there is another module inheriting from RetinaNet but not under models.detection(how likely is this?), the record of that module could be incorrect.
--EDIT--
Another disadvantage is that models inherit from TorchVision but not live in TorchVision will also be incorrectly logged as TorchVision events since the _log_api_usage_once adds torchvision. prefix to "models.detection"