Elerp

This WL script reads a MIDI file containing a sequence of K events (notes) and it creates an elastic path between K points in an N-Dimensional space (e.g. the latent space of a GAN). The returned sequence of vectors will be such that the corresponding outputs of the generator will form a movie with a swift dynamic near the MIDI events, and a slowly changing content in between. The final movie will be as if created at variable frame-rate, with accelerated and decelerated playback speed in pace with the MIDI track.

Usage

wolframscript -f Elerp.wl

You will be asked to insert the MIDI filename, a valid frame-rate r for the final video and the latent space dimension. Given that r cannot be smaller than the inverse of the minimum distance (in seconds) between two MIDI events, a non-valid frame-rate will be automatically replaced with the minimum valid one.

The output is a CSV file nameed elasticLatents.dat. Use this file as input of your generator!

Discussion

A linear interpolation between two latent vectors v₁ and v₂ can be written as

where 0 < f_t < 1 is a parametric curve of the parameter 0 < t < 1. In particular, for f_t = t we have the standard definition of lerp (linear interpolation).

An Elastic Lerp (Elerp) is a lerp defined by the following non-linear parametric curve:

that is a shifted and rescaled tangent with a time-warping parameter a. In the script, a is set to 0.77.

Note that the standard linear interpolation is the limit of the Elerp for zero time-warp.

The effect of the time-warping parameter is to slow down the pace of the interpolation between v₁ and v₂ and to accelerate it in their proximity.

Time Synchronization

Let's say that in the MIDI file there are K events occurring at t_k. In this case, we need to define K latent vectors v_k that are going to be interpolated. The script default is to generate K vectors with normal random components and order these vectors by finding the shortest tour. As an alternative, you can define or import your own latents.

Once we have all these K latents, we want the image generated from the v_k vector to appear at time t_k in the movie, the latent v_k+1 at time t_k+1 and so on. We achieve this by sampling the elastic path between v_k and v_k+1 with a number of interpolated vectors equal to r (t_k+1 - t_k)