Simple sin wave results

[huangtarn]

sinwave.csv
I am using a simple sin wave to test the algor. s1 is long wave, s2 is med wave, s3 is short wave, s4=s1+s2+s3. The model can predict s1/s2/s3 successfully, but for s4 it performs poorly compared to even LSTM. Could you share some insights on this?
I've tried default hyper-parameters, and also tried to tune it. No significant improvement.
thanks.

[huangtarn]

forgot to mention: sinwave.csv has s1/s2/s3/s4, I splatted them into 4 files for univar prediction.
total 10,000 points, s1 just 1 cycle, s2 50 cycles (1 cycles 200pnts), s3 500 cycles (20pnts).
prediction is on future 196 pnts.
The best s4 I can get is:
python train.py s4 s4 --archive forecast_csv_univar --batch-size 8 --repr-dim 320 --gpu 3 --eval --iters 1000

[zhihanyue]

I found that it did not perform well on both s1 and s4. In my experience, this is due to data drifting. Neural networks have a strong memory for input values. For example, in the following figure, if one trains the model on the left part and tests it on right part, the network may simply "memorize" the trend on "A" and output this trend for "B" (because the input values for "A" and "B" are very close). Therefore, in this kind of data with drifting, the neural networks without strong inductive bias can not be well generalized.

A workaround is to difference (y_t = x_t - x_{t-1}) the raw data to make it stationary. You can predict the differenced data y_t and then convert it back into prediction for raw data.

[huangtarn]

Zhihan, thanks for the tip. I guess there are two fundamental problems, the whole time series only contains about 1 cycle, may give a hard time for the algor to learn the long wave feature. 2. the instance is only 2, so probably the loss from instance is less significant compared to that of temporal loss.
But still a great implementation using the contrastive learning, it's fast too!