New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
More explicit gradient state interface #113
Conversation
Can we also rename the variable from Personally, in the context of a backwards pass, I feel that |
Thanks for bringing this up! I agree that Here's my proposal for names: export interface GradContext {
forward_inputs: [Tensor, ...ArgType[]]
forward_output: Tensor
backward_input: Tensor // the associated gradient of forward_output
backward_output_index: number // index of the associated forward input to be differentiated
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! one nit with the change to leakyRelu
@@ -16,7 +16,7 @@ export function relu(tensor: Tensor): Tensor { | |||
return tensor.maximum(scalar(0)) | |||
} | |||
|
|||
export function leakyRelu(tensor: Tensor, negative_slope: number): Tensor { | |||
export function leakyRelu(tensor: Tensor, negative_slope = 1e-3): Tensor { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fine to sneak this in, but can you annotate with the type? negative_slope: number = 1e-3
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
bun format
removes the type annotation when the arg has a default
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
:o @cryptodeal is that expected?
Clear naming is subjective, but tried to be more explicit.