Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add temporally consistent Query encodings #20

Closed
jacobbieker opened this issue Sep 28, 2021 · 0 comments · Fixed by #21
Closed

Add temporally consistent Query encodings #20

jacobbieker opened this issue Sep 28, 2021 · 0 comments · Fixed by #21
Labels
enhancement New feature or request

Comments

@jacobbieker
Copy link
Member

Detailed Description

From the discussion on #19 @JackKelly brought up:

I think the position encoding needs to be 'internally consistent' across each entire example. That is, the position encoding should be consistent across data inputs and the queries so the model can see that, say, the last timestep of the recent history in the input is immediately before the first timestep of the query. And that, in general, the model can see that the timeseries of recent history and the timeseries of queries are two parts of a contiguous timeseries.

To give a concrete example: If the recent history in the input spans 11:00 to 11:55, and the query is for 12:00 to 12:55, then we want the model to see that the first timestep of the query is 5 minutes after the last timestep of the input... if that makes sense?!

One way to do this might be to encode the positions once for all timesteps in the example (i.e. the concatenation of the recent history and the forecast timesteps), and then concatenate the last forecast_timesteps of the position encoding to the queries? Or something like that?! Not sure what's best!

Perhaps this could wait for a future PR though!

This is just a hunch, of course!

And, ultimately, we might want the position encoding to include both the relative position ("the ith element in the array") and the absolute position in time and space ("2pm in South London"). But we can worry about that later :)

Context

This could help the model with better understanding how the future timesteps encoded in the query relate to the input data.

Possible Implementation

Possibly by giving LearnableQuery the number of history timesteps, it could then add the features for the ones in the future? A better implementation might be to pass the Fourier encoding created across all the history and future input data, and then concatenate the future part to the learnable query.

@jacobbieker jacobbieker added the enhancement New feature or request label Sep 28, 2021
@jacobbieker jacobbieker added this to To do in National Grid Nowcasting: WP1 via automation Sep 28, 2021
@jacobbieker jacobbieker linked a pull request Oct 4, 2021 that will close this issue
7 tasks
@jacobbieker jacobbieker moved this from To do to In progress in National Grid Nowcasting: WP1 Oct 5, 2021
National Grid Nowcasting: WP1 automation moved this from In progress to Done Oct 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
No open projects
Development

Successfully merging a pull request may close this issue.

1 participant