Merging and pre hooks

[vshampor]

No description provided.

[vshampor]

This PR does two things:

1. Allows input tensors for NNCF-wrapped PyTorch functions to be pre-processed separately, e.g. introduces the "pre-hook" functionality; previously only the function output tensors could be post-processed.
2. Introduces a new algorithm for the merging of the propagating quantizers that have reached one and the same downward-branching node.

Will add more commentaries in-place.

[vshampor]

1. A PyTorch function input is not necessarily a single raw torch.Tensor for each input argument of the function. Consider torch.cat([x1, x2]), which has the input tensors x1 and x2 located inside a container, which itself is being passed as an argument. This function call, however, conceptually takes two tensors as its input, and if this is not traced properly, the NNCF graph will become disjoint. In a previous implementation this was circumvented by flattening the node inputs - a node's input signature, which identifies it among the other NNCF nodes, is a flat list of all tensors in the input arguments nesting hierarchy.

The same approach is taken here - a flat view of the function's input tensors is being constructed, and each tensor is conceptually assigned a separate "input port ID" ordinal. However, this flat view must be a view, i.e. it should allow the tensors to be replaced with the results of pre-processing while preserving the original nested structure. There should be an option to do so separately for each traced tensor present in the input argument container hierarchy - only in this case will the pre-hooking for separate inputs work in most cases. Sadly, Python has no real notion of writeable iterators; therefore, it is necessary for each tensor in a flat view to store and use associated getter and setter functions, which are built using the closest owning container's __getitem__ and __setitem__ and the associated index of the tensor in the said container.

[vshampor]

1. Since PyTorch/Python combo is essentially dynamic, there is no clear definition of how many inputs (in terms of NNCF graph nodes) does a given function function take (see torch.cat example above). The function's "input ports" therefore are not something static; rather, these are enumerated according to the number of traced tensors arriving as the function (node) input.

[vshampor]

1. An insertion point now stores an input port to affect, in order to represent the pre-hook semantics as well. None specifies a post-hook insertion.

[vshampor]

1. The InputAgnosticOperationExecutionContext is an address of a node, but the insertion of a hook for a node can now proceed in two ways - either post- or pre-. Therefore a position of a quantizer is no longer defined by a node, but by a node and the insertion type.

[vshampor]

1. Pre-hooks is what will allow to quantize branches in the network separately.
   Below, on the left, is the result of the MobileNetV2 quantization prior to this PR, and on the right - with this PR.
   

Notice that the __add__ node used to have a single, nominal pre-hook insertion point, while in fact it takes two tensor activations. The pre-hook insertion points were also non-functional, i.e. the actual insertion could only occur at a post-hook point, which led to a situation where either both of the branches would be quantized with a single quantizer (if a FQ is inserted in the top-most post-hook insertion point), or none of them would.

With the PR, the node has two pre-hook insertion points, corresponding to the number of inputs (i.e. "ports"). Furthermore, it is possible to separately quantize the conv2d input, using the conv2d's pre-hook 0, and the residual __add__ input, using the __add__'s pre-hook 0.

[vshampor]

2. The quantizer propagation algorithm has been modified to work in two stages. This is related to the new quantizer configuration merging algorithm. Basically the first step of the propagation results in a proposal - a set of the points at which a quantizer should be inserted in order to quantize all operations specified in HW config with a supported configuration, and, for each point, a list of possible quantizer configurations out of which one and only one should be chosen. The choice could be either affected by utilizing the NNCF config options/presets, or it could be delegated to a specialized algorithm, such as HAWQ in case multiple bitwidths are present in the quantizer configuration list, or an RL algorithm.

Once the final quantizer configurations are chosen, an extra step is necessary to merge the quantizers that are redundant - e.g. if one and the same final quantizer configuration has been chosen both for the global quantizer of all downstream branches and for the local quantizer on a branch. This is illustrated below - at the top is the "proposed" state, which allows a choice among multiple quantizer configuration for each quantizer (blue nodes). At the bottom is the final state provided that the user's/algorithm's choice for quantizers 59, 7 and 4 had all been to use 8 bit symmetric per-tensor quantization; the quantizers 7 and 4 are redundant with respect to quantizer 59 and therefore are discarded.

[vshampor]

The AGGRESSIVE strategy will be used by default, and it is the one most likely to garner performance benefits.

[vshampor]

Now that all insertion points, both pre- and post-hook, are functional, this function is mostly obsolete.

[vshampor]

2. The new propagation algorithm will make all quantizers that are trying to pass up through a downward-branching node wait until the quantizers on the neighbouring branches reach that node as well. Only then will the merge process occur, which may result in the new merge quantizer being added above the branching node to propagate further, and in the existing branch quantizers being discarded. For the PropagationStrategy.AGGRESSIVE, it is also possible that neither of the branch quantizers is discarded, but the merge quantizer is still created - for instance, if one of the branches will potentially require requantization, in case the user/specialized algorithm chooses a corresponding configuration.

Here, the merge quantizer will have a single 8-bit configuration associated since this is the only option that is supported by all the downstream branches. If, at the final quantization config selection stage of the setup, some of the branches also end up having 8-bit quantization chosen for them, the corresponding branch quantizers will be discarded since the global quantizer already has 8-bit quantization. The user is still, however, presented with a possibility to choose non-8-bit quantization for all the branches, in which case requantization will occur on the branches.

[vshampor]

The string representation of a quantizer has changed since now it needs to reflect the insertion type - either as an input quantizer to an operation or as an output quantizer. This may affect the code that relies upon it - please comment if you know of such pieces of code.

[vshampor]

@alexsu52 @AlexKoff88 @ljaljushkin @asenina, add others as necessary.

[asenina]

Jenkins please retry a build

[ljaljushkin]

This graph looks strange now

1. input for AvgPool can't be INT4 by HW config.
2. previously, we had the same quantization parameters on all inputs and outputs of the concat node, which is HW-friendly. now, we allow uncoupled bitwidth of activations on the branches.

how it was before

[vshampor]

After enabling heterogenous quantizer configuration handling for HAWQ, I managed to restore most of the reference bitwidth distribution graphs to the same state that is currently present on develop.
Current state of the same piece of the graph that we considered in this discussion:

Note that 4-bit AFQ requantizations appeared on the branches. This is due to the fact top-level FQ is 8-bit asymmetric per-tensor, which is the only configuration compatible with both the 8-bit symmetric per-channel listed as supported by avg_pool2d in vpu.json, and the conv2d inputs, which support 4-bit symmetric per-tensor and 8-bit asymmetric per-tensor quantizations. Since 4 bits have been chosen for conv2d weights, the corresponding activation FQs have been selected as 4-bit, which leads to requantization. Another requantization (no. 104) occurs for avg_pool2d to narrow the quantized output range from asymmetric to symmetric; although per-channel quantization in no.104 is not narrowing w.r.t. nos. 34, 45, 61, 69, this does not break anything.

Also note that the requantizers share the adjacency group with their base quantizers. Hope this is OK.

[vshampor]

Jenkins please retry a build