Speed of profiling

[nasretdinovr]

It takes about 30min to 1 hour to profile my network (transformer-like architecture, about 20 million parameters). Is it normal or I do something wrong?

[gaoteng-git]

@nasretdinovr How many steps do you profile? What's the arguments of torch.profiler.profile? Would you please share your profiling code here?

[nasretdinovr]

@gaoteng-git I took the code from ResNet example in PyTorch doc and adapted it for my network. To profile ResNet it takes about 5 min.

with torch.profiler.profile(
     activities=[
        torch.profiler.ProfilerActivity.CPU,
        torch.profiler.ProfilerActivity.CUDA],
    schedule=torch.profiler.schedule(
        wait=2,
        warmup=3,
        active=6),
    on_trace_ready=torch.profiler.tensorboard_trace_handler('./result', worker_name='worker0'),
    record_shapes=True,
    profile_memory=True,
    with_stack=True
) as p:
    for batch_idx, data in enumerate(train_loader, 0):
         print("step:{}".format(batch_idx))

        spectrograms, labels, input_lengths, label_lengths = data
        spectrograms, labels = spectrograms[:, :, :, :max(input_lengths)].to(device), labels.to(
            device)  # (batch, 1, feat_dim, time)
        spectrograms = spectrograms.squeeze(1).transpose(1, 2)  # (batch, time, feat_dim,)
        optimizer.zero_grad()

        output = model(spectrograms, input_lengths)  # (batch, time, n_classes)
        output = F.log_softmax(output, dim=2)
        output = output.transpose(0, 1)  # (time, batch, n_class)
        input_lengths = [x // 4 for x in input_lengths]
        loss = criterion(output, labels, input_lengths, label_lengths)
        loss.backward()

        torch.nn.utils.clip_grad_norm_(model.parameters(), 5.0)
        optimizer.step()

        if batch_idx + 1 >= 22:
            break
        p.step()

[gaoteng-git]

@nasretdinovr Does it hang at "step:10" and "step:21"?
You could try to disable "profile_memory" by setting it to False or just removing it.
The current PyTorch profiler's memory events handling is low efficient and very slow. We will contact Facebook to optimize it.
Sorry that we didn't notice the example code will mislead user. This is a quick fix on the example code: #316

[nasretdinovr]

@gaoteng-git Disabling "profile_memory" really helped! Thank you a lot!

[guotuofeng]

@gdankel, do we need remove the legacy events from kineto events? https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/profiler_kineto.cpp#L130

[gdankel]

Not entirely sure - let's check with @ilia-cher