New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ignore no-gradient functions for "Jacobian needed" warning when sampling transformed vars #228

Closed
bob-carpenter opened this Issue Sep 20, 2013 · 12 comments

Comments

Projects
None yet
4 participants
@bob-carpenter
Contributor

bob-carpenter commented Sep 20, 2013

Indexing functions, row-extraction functions, etc.

  • create singleton symbol table to store them along same lines as function_signatures and reserved words
  • go through functions and add relevant ones
  • write visitor in AST code to walk through expression and make sure only kosher functions are used, and if not, raise the warning
@syclik

This comment has been minimized.

Show comment
Hide comment
@syclik

syclik Nov 1, 2013

Member

#294 duplicates this one.

Member

syclik commented Nov 1, 2013

#294 duplicates this one.

@syclik

This comment has been minimized.

Show comment
Hide comment
@syclik

syclik May 5, 2014

Member

I just verified the behavior on this one. I think it should be marked as closed. Thanks for getting it done!

Perhaps a new one should be opened: I still get the warning for transformed parameters. For example:

parameters {
  real x;
}
transformed parameters {
  real y;
  y <- x;
}
model {
  y ~ normal(0, 5);
}

gives me this warning:

Warning (non-fatal): Left-hand side of sampling statement (~) contains a non-linear transform of a parameter or local variable.
 You must call increment_log_prob() with the log absolute determinant of the Jacobian of the transform.
  Sampling Statement left-hand-side expression:
    y ~ normal_log(...)
Member

syclik commented May 5, 2014

I just verified the behavior on this one. I think it should be marked as closed. Thanks for getting it done!

Perhaps a new one should be opened: I still get the warning for transformed parameters. For example:

parameters {
  real x;
}
transformed parameters {
  real y;
  y <- x;
}
model {
  y ~ normal(0, 5);
}

gives me this warning:

Warning (non-fatal): Left-hand side of sampling statement (~) contains a non-linear transform of a parameter or local variable.
 You must call increment_log_prob() with the log absolute determinant of the Jacobian of the transform.
  Sampling Statement left-hand-side expression:
    y ~ normal_log(...)
@syclik

This comment has been minimized.

Show comment
Hide comment
@syclik

syclik May 5, 2014

Member

That is, I verified it with the current develop branch as of 5/5/2014.

Member

syclik commented May 5, 2014

That is, I verified it with the current develop branch as of 5/5/2014.

@bob-carpenter

This comment has been minimized.

Show comment
Hide comment
@bob-carpenter

bob-carpenter May 5, 2014

Contributor

Should we just get rid of this warning altogether?
Or should I qualify it further?

I've already set it up now so that some of the true positives
and now false negatives to cut down on false positives.

I just flag every transformed parameter or local variable that's
used because otherwise I'd have to put a history on every variable
and infer whether or not a non-linear function was involved.

  • Bob

On May 5, 2014, at 2:25 PM, Daniel Lee notifications@github.com wrote:

I just verified the behavior on this one. I think it should be marked as closed. Thanks for getting it done!

Perhaps a new one should be opened: I still get the warning for transformed parameters. For example:

parameters {
real x;
}
transformed parameters {
real y;
y <- x;
}
model {
y ~ normal(0, 5);
}

gives me this warning:

Warning (non-fatal): Left-hand side of sampling statement (~) contains a non-linear transform of a parameter or local variable.
You must call increment_log_prob() with the log absolute determinant of the Jacobian of the transform.
Sampling Statement left-hand-side expression:
y ~ normal_log(...)


Reply to this email directly or view it on GitHub.

Contributor

bob-carpenter commented May 5, 2014

Should we just get rid of this warning altogether?
Or should I qualify it further?

I've already set it up now so that some of the true positives
and now false negatives to cut down on false positives.

I just flag every transformed parameter or local variable that's
used because otherwise I'd have to put a history on every variable
and infer whether or not a non-linear function was involved.

  • Bob

On May 5, 2014, at 2:25 PM, Daniel Lee notifications@github.com wrote:

I just verified the behavior on this one. I think it should be marked as closed. Thanks for getting it done!

Perhaps a new one should be opened: I still get the warning for transformed parameters. For example:

parameters {
real x;
}
transformed parameters {
real y;
y <- x;
}
model {
y ~ normal(0, 5);
}

gives me this warning:

Warning (non-fatal): Left-hand side of sampling statement (~) contains a non-linear transform of a parameter or local variable.
You must call increment_log_prob() with the log absolute determinant of the Jacobian of the transform.
Sampling Statement left-hand-side expression:
y ~ normal_log(...)


Reply to this email directly or view it on GitHub.

@syclik syclik closed this May 15, 2014

@bob-carpenter

This comment has been minimized.

Show comment
Hide comment
@bob-carpenter

bob-carpenter May 15, 2014

Contributor

I'm in favor of just getting rid of this and all the infrastructure associated with it.

Contributor

bob-carpenter commented May 15, 2014

I'm in favor of just getting rid of this and all the infrastructure associated with it.

@syclik

This comment has been minimized.

Show comment
Hide comment
@syclik

syclik May 15, 2014

Member

I second that.

Member

syclik commented May 15, 2014

I second that.

@mbrubake

This comment has been minimized.

Show comment
Hide comment
@mbrubake

mbrubake May 16, 2014

Member

I third it. It would be nice if we could do it, but getting it right all
the time will be hard and getting it wrong most of the time is just causing
unnecessary stress among users.

On Thu, May 15, 2014 at 5:05 PM, Daniel Lee notifications@github.comwrote:

I second that.


Reply to this email directly or view it on GitHubhttps://github.com//issues/228#issuecomment-43264883
.

Member

mbrubake commented May 16, 2014

I third it. It would be nice if we could do it, but getting it right all
the time will be hard and getting it wrong most of the time is just causing
unnecessary stress among users.

On Thu, May 15, 2014 at 5:05 PM, Daniel Lee notifications@github.comwrote:

I second that.


Reply to this email directly or view it on GitHubhttps://github.com//issues/228#issuecomment-43264883
.

@bentleygcoffey

This comment has been minimized.

Show comment
Hide comment
@bentleygcoffey

bentleygcoffey Aug 7, 2014

I'm one of those users confused by the warning for transformed parameters... How do I determine whether my transformation is kosher (i.e. a false positive) or somehow inappropriate (i.e. a true positive)? An analogous warning was thrown for me when I tested the model with the reparameterized beta on page 155 of the reference (version 2.2.0):

image
image

bentleygcoffey commented Aug 7, 2014

I'm one of those users confused by the warning for transformed parameters... How do I determine whether my transformation is kosher (i.e. a false positive) or somehow inappropriate (i.e. a true positive)? An analogous warning was thrown for me when I tested the model with the reparameterized beta on page 155 of the reference (version 2.2.0):

image
image

@bentleygcoffey

This comment has been minimized.

Show comment
Hide comment
@bentleygcoffey

bentleygcoffey Aug 7, 2014

Gives me this warning:

DIAGNOSTIC(S) FROM PARSER:

Warning (non-fatal): Left-hand side of sampling statement (~) contains a non-linear transform of a parameter or local variable.
You must call increment_log_prob() with the log absolute determinant of the Jacobian of the transform.

bentleygcoffey commented Aug 7, 2014

Gives me this warning:

DIAGNOSTIC(S) FROM PARSER:

Warning (non-fatal): Left-hand side of sampling statement (~) contains a non-linear transform of a parameter or local variable.
You must call increment_log_prob() with the log absolute determinant of the Jacobian of the transform.

@bob-carpenter

This comment has been minimized.

Show comment
Hide comment
@bob-carpenter

bob-carpenter Aug 7, 2014

Contributor

A transform is kosher if the log Jacobian determinant of
the transform is constant (i.e., doesn't depend on any
parameters). In the univariate case, that means its derivative
is constant and doesn't depend on any parameters (e.g.,
2*x has a constant derivative but exp(x) does not).

Transforms also have to be differentiable and monotonic.

You didn't provide enough context to diagnose what's going on
in your case --- none of those lines should cause a problem
in Stan 2.4. f you want more specific help, please send the full model
(or even better, a small model with the same issue), along with
the full error message including line numbers, so we can see where the
error's at.

  • Bob

On Aug 7, 2014, at 10:49 AM, bentleygcoffey notifications@github.com wrote:

I'm one of those users confused by the warning for transformed parameters... How do I determine whether my transformation is kosher (i.e. a false positive) or somehow inappropriate (i.e. a true positive)? An analogous warning was thrown for me when I tested the model with the reparameterized beta on page 155 of the reference (version 2.2.0):


Reply to this email directly or view it on GitHub.

Contributor

bob-carpenter commented Aug 7, 2014

A transform is kosher if the log Jacobian determinant of
the transform is constant (i.e., doesn't depend on any
parameters). In the univariate case, that means its derivative
is constant and doesn't depend on any parameters (e.g.,
2*x has a constant derivative but exp(x) does not).

Transforms also have to be differentiable and monotonic.

You didn't provide enough context to diagnose what's going on
in your case --- none of those lines should cause a problem
in Stan 2.4. f you want more specific help, please send the full model
(or even better, a small model with the same issue), along with
the full error message including line numbers, so we can see where the
error's at.

  • Bob

On Aug 7, 2014, at 10:49 AM, bentleygcoffey notifications@github.com wrote:

I'm one of those users confused by the warning for transformed parameters... How do I determine whether my transformation is kosher (i.e. a false positive) or somehow inappropriate (i.e. a true positive)? An analogous warning was thrown for me when I tested the model with the reparameterized beta on page 155 of the reference (version 2.2.0):


Reply to this email directly or view it on GitHub.

@bentleygcoffey

This comment has been minimized.

Show comment
Hide comment
@bentleygcoffey

bentleygcoffey Aug 7, 2014

Yeah, I did play around with that code a bit. The theta[n], which is the data in that example from the reference, was defined locally in my model section as the quotient of one variable divided by the other within the model section (in the line prior to the sampling statement of that quotient's likelihood). I'm guessing that the code just goes through the model statement and warns if it finds any transformation of the data on the LHS of a sampling statement, even if it didn't even involve a parameter (in which case the derivative with respect to a parameter is 0). So I guess that once the user determines that the transformation is Kosher, the user can ignore the warning and/or (when possible) perform any transformation prior to calling stan (in my case, from r)? If so, perhaps the warning could say something to that effect:

In addition to being monotonic and differentiable, the transform of the variable before the ~ must be invariant with respect to the parameters to be estimated, in which case this warning can be ignored, or else the transform's log Jacobian determinant must be accounted for in the model block via a call to increment_log_prob()

bentleygcoffey commented Aug 7, 2014

Yeah, I did play around with that code a bit. The theta[n], which is the data in that example from the reference, was defined locally in my model section as the quotient of one variable divided by the other within the model section (in the line prior to the sampling statement of that quotient's likelihood). I'm guessing that the code just goes through the model statement and warns if it finds any transformation of the data on the LHS of a sampling statement, even if it didn't even involve a parameter (in which case the derivative with respect to a parameter is 0). So I guess that once the user determines that the transformation is Kosher, the user can ignore the warning and/or (when possible) perform any transformation prior to calling stan (in my case, from r)? If so, perhaps the warning could say something to that effect:

In addition to being monotonic and differentiable, the transform of the variable before the ~ must be invariant with respect to the parameters to be estimated, in which case this warning can be ignored, or else the transform's log Jacobian determinant must be accounted for in the model block via a call to increment_log_prob()

@bob-carpenter

This comment has been minimized.

Show comment
Hide comment
@bob-carpenter

bob-carpenter Aug 7, 2014

Contributor

It always flags local variables, because it can't guarantee
their provenance. You'd need to track back through the execution,
and it's not even a decidable problem.

Transformation of data is OK. What is flagged is uses of
parameters, transformed parameters, or local variables in
the model block which can't be guaranteed to be linear. So
there are definitely false positivies.

I think that's a good suggestion for the message. We're trying
to do a pass and clean up all of our error messages to make them
more understandable. I created the following issue for the change:

#831

  • Bob

On Aug 7, 2014, at 12:17 PM, bentleygcoffey notifications@github.com wrote:

Yeah, I did play around with that code a bit. The theta[n], which is the data in that example from the reference, was defined locally in my model section as the quotient of one variable divided by the other within the model section (in the line prior to the sampling statement of that quotient's likelihood). I'm guessing that the code just goes through the model statement and warns if it finds any transformation of the data on the LHS of a sampling statement, even if it didn't even involve a parameter (in which case the derivative with respect to a parameter is 0). So I guess that once the user determines that the transformation is Kosher, the user can ignore the warning and/or (when possible) perform any transformation prior to calling stan (in my case, from r)? If so, perhaps the warning could say something to that effect:

In addition to being monotonic and differentiable, the transform of the variable before the ~ must be invariant with respect to the parameters to be estimated, in which case this warning can be ignored, or else the transform's log Jacobian determinant must be accounted for in the model block via a call to increment_log_prob()


Reply to this email directly or view it on GitHub.

Contributor

bob-carpenter commented Aug 7, 2014

It always flags local variables, because it can't guarantee
their provenance. You'd need to track back through the execution,
and it's not even a decidable problem.

Transformation of data is OK. What is flagged is uses of
parameters, transformed parameters, or local variables in
the model block which can't be guaranteed to be linear. So
there are definitely false positivies.

I think that's a good suggestion for the message. We're trying
to do a pass and clean up all of our error messages to make them
more understandable. I created the following issue for the change:

#831

  • Bob

On Aug 7, 2014, at 12:17 PM, bentleygcoffey notifications@github.com wrote:

Yeah, I did play around with that code a bit. The theta[n], which is the data in that example from the reference, was defined locally in my model section as the quotient of one variable divided by the other within the model section (in the line prior to the sampling statement of that quotient's likelihood). I'm guessing that the code just goes through the model statement and warns if it finds any transformation of the data on the LHS of a sampling statement, even if it didn't even involve a parameter (in which case the derivative with respect to a parameter is 0). So I guess that once the user determines that the transformation is Kosher, the user can ignore the warning and/or (when possible) perform any transformation prior to calling stan (in my case, from r)? If so, perhaps the warning could say something to that effect:

In addition to being monotonic and differentiable, the transform of the variable before the ~ must be invariant with respect to the parameters to be estimated, in which case this warning can be ignored, or else the transform's log Jacobian determinant must be accounted for in the model block via a call to increment_log_prob()


Reply to this email directly or view it on GitHub.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment