Signatory slower than iisignature

[phaelicks]

Dear Patrick,

first, thank you for this great package and its documentation! I am following up on the comment in the documentation's FAQs, that Signatory should be faster than iisignature even on CPU. https://signatory.readthedocs.io/en/latest/pages/miscellaneous/faq.html

I wrote a simple script to compare the time needed for calculating the signature of a path of both packages. Turns out, for shorter paths (up to length of 10^3 or 10^4), a smaller signature truncation depth (up to level 3 or 4), as well as small number of input channels (up to 6 or 7) the iisignature package is substantially faster. Depending on the values of above three parameters between 1.5 and 12 times faster, whereas the speed up decreases the larger the parameter values.

This stands partly in contrast to the claim in the package's documentation. Do you have an explanation why signatory is slower? Is there a way to speed up calculations for shorter paths?

Specifically: The function 'signature' provides the functionality of updating the signature of a path with new information using Chen's idendity, which should be faster than computing the whole signature again. Does this functionality work with automatic gradient calculations?

I ran the script on CPU on a 2,6 GHz Quad-Core Intel Core i7 MacBook Pro from 2016 with the following versions:

• Python: 3.8
• signatory: 1.2.3 (in combination with torch 1.6.0)
• iisignature: 0.24

Some background: I am currently using signatory in a reinforcement learning project, utilizing its implementation of the signature calculation as PyTorch nn.Module for automatic gradient calculations, however training my algorithm takes a very long time, since the signature is calculsted at every step the RL algorithm takes. I have been working with the iisignature package before, but switched to your signatory package to be able to fully use automatic differentiation (I am still new to PyTorch, so the signatory implementation was great help). The paper "Deep Signature Transform" was part of the inspiration for my project. Trying to reproduce some results of the paper, I realized how much faster computations where, which led me to this simple experiment.

I include my script for time comparison below.
Thank you and best, Felix

import torch
import signatory
import iisignature
import time
import numpy as np

runs = 10
max_length = 6
channels = 7
sig_depth = 3
times_iisig = np.zeros(max_length)
times_signatory = np.zeros(max_length)

for j in range(runs):
    for i in range(max_length):
        path = torch.rand(10**(i+1), channels)

        start = time.time()
        sig = iisignature.sig(path, sig_depth)
        end = time.time()
        times_iisig[i] += (end - start)    
        del(sig)

        path = path.unsqueeze(0)
        start = time.time()
        sig = signatory.signature(path, sig_depth)
        end = time.time()
        times_signatory[i] += (end - start)
        del(path, sig)

print('Times iisignsture: \n', times_iisig / runs)
print('Times signatory: \n ', times_signatory / runs)
print('Speed up: \n', times_signatory / times_iisig)
print('Average speed up:', (times_signatory / times_iisig).mean())

[patrick-kidger]

Hi there!

It's not immediately clear to me what's going on here. One thing you could try is running Signatory's own benchmarks: python command.py benchmark ... and see what you get.

It's possible that iisignature has been updated since Signatory was first written, and it might be doing something smarter now. In particular Signatory introduced a new asymptotically faster algorithm for computing signatures, and this was the main reason for its speedup -- perhaps iisignature is now using that too?

In any case, I should be honest that at this point Signatory isn't really maintained, and there probably isn't much I can do to help.

[louisamand]

On a single path iisignature is faster (probably because of the overhead of using PyTorch), but I believe that for multiple paths (i.e.shape=(num_sim, time_steps, channels)) it is much, much quicker, taking advantage of parallelization and GPU.

[phaelicks]

Hi,
thank you for the replies!

Indeed, running tests on multiple paths and / or higher number of input channels the advantage of signatory becomes obvious. Pity, that signatory is not maintained anymore, it is a great package!