[fx/profiler] provide a table of summary.

[super-dainiu]

What's new?

I create a summary method for MetaInfoProp. Hopefully, this will help @Cypher30 debug backward memory estimations.

BATCH_SIZE = 2
DIM_IN = 4
DIM_HIDDEN = 16
DIM_OUT = 16
model = torch.nn.Sequential(
          torch.nn.Linear(DIM_IN, DIM_HIDDEN), 
          torch.nn.Linear(DIM_HIDDEN, DIM_OUT),
          )
input_sample = torch.rand(BATCH_SIZE, DIM_IN)
gm = symbolic_trace(model)
interp = MetaInfoProp(gm)
interp.run(input_sample)
print(interp.summary(format='kb'))    # don't panic if some statistics are 0.00 MB

======================================================
# output of above code is 
    Op type       Op    Forward FLOPs    Backward FLOPs    SAVE_FWD_IN    FWD_OUT    FWD_TMP    BWD_OUT    BWD_TMP
-----------  -------  ---------------  ----------------  -------------  ---------  ---------  ---------  ---------
placeholder  input_1          0 FLOPs           0 FLOPs          False    0.00 KB    0.00 KB    0.00 KB    0.00 KB
call_module       _0        128 FLOPs         288 FLOPs           True    0.12 KB    0.00 KB    0.34 KB    0.00 KB
call_module       _1        512 FLOPs       1,056 FLOPs           True    0.12 KB    0.00 KB    1.19 KB    0.00 KB
     output   output          0 FLOPs           0 FLOPs           True    0.00 KB    0.00 KB    0.00 KB    0.00 KB

Warnings

@Cypher30
Obviously, nn.Linear's BWD_OUT should be the same as the previous node's FWD_OUT.

It is fairly likely that BWD_OUT also includes a gradient for weight and bias.

[super-dainiu]

we need at least run meta once before getting this summary

[FrankLeeeee]

should it be unit instead of format?

[super-dainiu]

yes