functional/functor testing framework for first- and higher-order autodiff

[bob-carpenter]

To remove duplication from our derivative testing code and also make it portable for when we upgrade autodiff, we need to move our testing into a generic functional framework that takes pairs of functors and compares them.

[betanalpha]

Stan-devs should check out my attempt at this in the Nomad repo.

On Nov 19, 2014, at 7:04 PM, Bob Carpenter notifications@github.com wrote:

To remove duplication from our derivative testing code and also make it portable for when we upgrade autodiff, we need to move our testing into a generic functional framework that takes pairs of functors and compares them.

—
Reply to this email directly or view it on GitHub.

[bob-carpenter]

It'd be nice if the tests were factored so that the
tests for functions didn't depend on the details of
the Nomad data structure.

Rob and I just started doing this on a branch
for Stan, and if you have the autodiff.hpp implemented,
it should port to Nomad.

• Bob

On Nov 19, 2014, at 3:17 PM, Michael Betancourt notifications@github.com wrote:

Stan-devs should check out my attempt at this in the Nomad repo.

On Nov 19, 2014, at 7:04 PM, Bob Carpenter notifications@github.com wrote:

To remove duplication from our derivative testing code and also make it portable for when we upgrade autodiff, we need to move our testing into a generic functional framework that takes pairs of functors and compares them.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub.

[betanalpha]

What do you mean by Nomad data structure?
The test pattern in Nomad should work for any functor/functional setup, no?

On Nov 19, 2014, at 9:09 PM, Bob Carpenter notifications@github.com wrote:

It'd be nice if the tests were factored so that the
tests for functions didn't depend on the details of
the Nomad data structure.

Rob and I just started doing this on a branch
for Stan, and if you have the autodiff.hpp implemented,
it should port to Nomad.

• Bob

On Nov 19, 2014, at 3:17 PM, Michael Betancourt notifications@github.com wrote:

Stan-devs should check out my attempt at this in the Nomad repo.

On Nov 19, 2014, at 7:04 PM, Bob Carpenter notifications@github.com wrote:

To remove duplication from our derivative testing code and also make it portable for when we upgrade autodiff, we need to move our testing into a generic functional framework that takes pairs of functors and compares them.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub.

[bob-carpenter]

I don't have the Nomad test code in front of me, but when Rob was
showing it to me, tests for functions involved the use of
Nomad data structures.

I think it'll be easiest and most portable if the tests
only need to talk about our functions and double values.

This is the same point I was trying to make about defining
functions using structs. If I remember correctly, Michael
completely disagreed, and the result is that definining functions
in Nomad requires tweaking Nomad data structures [as it does
in Stan 2.5].

Here's what the test for log1p looks like in the framework
we just roughed out (so far only testing first order, but
the call will be exactly the same when the rest is filled in):

#include <test/unit/agrad/agrad-functor-eq.hpp>

struct log1p_primitive {
  template <typename T>
  T operator()(const T& x) const {
    using ::log;
    return log(1 + x);
  }
};

struct log1p_stan {
  template <typename T>
  T operator()(const T& x) const {
    using stan::math::log1p;
    return log1p(x);
  }
};

TEST(stanMath, log1p_derivs) {
  using test_unit::expect_agrad_eq_unary;
  log1p_primitive f1;
  log1p_stan f2;
  expect_agrad_eq_unary(f1,f2,0.3);
}

// you would use this if log1p were defined to throw, but it's not

// TEST(stanMath, log1p_throws) {
//   using test_unit::expect_throws_unary;
//   log1p_stan f;

//   expect_throws_unary<std::domain_error>(f,-2.7);
// }

• Bob

On Nov 19, 2014, at 4:22 PM, Michael Betancourt notifications@github.com wrote:

What do you mean by Nomad data structure?
The test pattern in Nomad should work for any functor/functional setup, no?

On Nov 19, 2014, at 9:09 PM, Bob Carpenter notifications@github.com wrote:

It'd be nice if the tests were factored so that the
tests for functions didn't depend on the details of
the Nomad data structure.

Rob and I just started doing this on a branch
for Stan, and if you have the autodiff.hpp implemented,
it should port to Nomad.

• Bob

On Nov 19, 2014, at 3:17 PM, Michael Betancourt notifications@github.com wrote:

Stan-devs should check out my attempt at this in the Nomad repo.

On Nov 19, 2014, at 7:04 PM, Bob Carpenter notifications@github.com wrote:

To remove duplication from our derivative testing code and also make it portable for when we upgrade autodiff, we need to move our testing into a generic functional framework that takes pairs of functors and compares them.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub.

[betanalpha]

https://github.com/stan-dev/nomad/blob/master/src/test/scalar/functions/smooth_functions/acos_test.cpp
https://github.com/stan-dev/nomad/blob/master/src/test/finite_difference.hpp

On Nov 19, 2014, at 11:15 PM, Bob Carpenter notifications@github.com wrote:

I don't have the Nomad test code in front of me, but when Rob was
showing it to me, tests for functions involved the use of
Nomad data structures.

I think it'll be easiest and most portable if the tests
only need to talk about our functions and double values.

This is the same point I was trying to make about defining
functions using structs. If I remember correctly, Michael
completely disagreed, and the result is that definining functions
in Nomad requires tweaking Nomad data structures [as it does
in Stan 2.5].

Here's what the test for log1p looks like in the framework
we just roughed out (so far only testing first order, but
the call will be exactly the same when the rest is filled in):

#include <test/unit/agrad/agrad-functor-eq.hpp> 

struct log1p_primitive { 
template <typename T> 
T operator()(const T& x) const { 
using ::log; 
return log(1 + x); 
} 
}; 

struct log1p_stan { 
template <typename T> 
T operator()(const T& x) const { 
using stan::math::log1p; 
return log1p(x); 
} 
}; 

TEST(stanMath, log1p_derivs) { 
using test_unit::expect_agrad_eq_unary; 
log1p_primitive f1; 
log1p_stan f2; 
expect_agrad_eq_unary(f1,f2,0.3); 
} 

// you would use this if log1p were defined to throw, but it's not 

// TEST(stanMath, log1p_throws) { 
// using test_unit::expect_throws_unary; 
// log1p_stan f; 

// expect_throws_unary<std::domain_error>(f,-2.7); 
// } 

• Bob

On Nov 19, 2014, at 4:22 PM, Michael Betancourt notifications@github.com wrote:

What do you mean by Nomad data structure?
The test pattern in Nomad should work for any functor/functional setup, no?

On Nov 19, 2014, at 9:09 PM, Bob Carpenter notifications@github.com wrote:

It'd be nice if the tests were factored so that the
tests for functions didn't depend on the details of
the Nomad data structure.

Rob and I just started doing this on a branch
for Stan, and if you have the autodiff.hpp implemented,
it should port to Nomad.

• Bob

On Nov 19, 2014, at 3:17 PM, Michael Betancourt notifications@github.com wrote:

Stan-devs should check out my attempt at this in the Nomad repo.

On Nov 19, 2014, at 7:04 PM, Bob Carpenter notifications@github.com wrote:

To remove duplication from our derivative testing code and also make it portable for when we upgrade autodiff, we need to move our testing into a generic functional framework that takes pairs of functors and compares them.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub.

[bob-carpenter]

Cool --- this is great. It's a more mature version of exactly what
I was talking about! Partly the rest of this is notes for Rob with
some suggestions embedded.

What we should do is refactor what we have in Stan to use the Nomad
testing framework.

I like that it's done on vectors, but Rob and I hadn't
figured out how to resolve issues of mixed double and
var inputs. I see that you've pushed that up to the
test functor, which is a neat solution:

https://github.com/stan-dev/nomad/blob/master/src/test/scalar/functions/smooth_functions/atan2_test.cpp

Then you need to define multiple functors. I don't offhand
see a way that you could just define a single templated structure,
but it might be use putting some thought into.

You could avoid the hard coding of a single test value by adding the
double value as a member variable.

Are you doing the double-double tests elsewhere? I see that you don't have
a using statement in the test functors (it just uses argument-dependent
lookup).

I also like the addition of the name to the functor.

You could get around the virtuality of base_functor::operator() if you
use the "curiously recursive template pattern" --- it's what Eigen does
for its indexing operator overrides. No big deal if it's only
used in tests, though I see it's defined outside the test framework.

I keep thinking that function pointers could get rid of the need to
define all the functors wrapping functions like acos, but the
problem is that you need the template instantiation.

• Bob

On Nov 19, 2014, at 7:09 PM, Michael Betancourt notifications@github.com wrote:

https://github.com/stan-dev/nomad/blob/master/src/test/scalar/functions/smooth_functions/acos_test.cpp
https://github.com/stan-dev/nomad/blob/master/src/test/finite_difference.hpp

On Nov 19, 2014, at 11:15 PM, Bob Carpenter notifications@github.com wrote:

I don't have the Nomad test code in front of me, but when Rob was
showing it to me, tests for functions involved the use of
Nomad data structures.

I think it'll be easiest and most portable if the tests
only need to talk about our functions and double values.

This is the same point I was trying to make about defining
functions using structs. If I remember correctly, Michael
completely disagreed, and the result is that definining functions
in Nomad requires tweaking Nomad data structures [as it does
in Stan 2.5].

Here's what the test for log1p looks like in the framework
we just roughed out (so far only testing first order, but
the call will be exactly the same when the rest is filled in):

#include <test/unit/agrad/agrad-functor-eq.hpp> 

struct log1p_primitive { 
template <typename T> 
T operator()(const T& x) const { 
using ::log; 
return log(1 + x); 
} 
}; 

struct log1p_stan { 
template <typename T> 
T operator()(const T& x) const { 
using stan::math::log1p; 
return log1p(x); 
} 
}; 

TEST(stanMath, log1p_derivs) { 
using test_unit::expect_agrad_eq_unary; 
log1p_primitive f1; 
log1p_stan f2; 
expect_agrad_eq_unary(f1,f2,0.3); 
} 

// you would use this if log1p were defined to throw, but it's not 

// TEST(stanMath, log1p_throws) { 
// using test_unit::expect_throws_unary; 
// log1p_stan f; 

// expect_throws_unary<std::domain_error>(f,-2.7); 
// } 

• Bob

On Nov 19, 2014, at 4:22 PM, Michael Betancourt notifications@github.com wrote:

What do you mean by Nomad data structure?
The test pattern in Nomad should work for any functor/functional setup, no?

On Nov 19, 2014, at 9:09 PM, Bob Carpenter notifications@github.com wrote:

It'd be nice if the tests were factored so that the
tests for functions didn't depend on the details of
the Nomad data structure.

Rob and I just started doing this on a branch
for Stan, and if you have the autodiff.hpp implemented,
it should port to Nomad.

• Bob

On Nov 19, 2014, at 3:17 PM, Michael Betancourt notifications@github.com wrote:

Stan-devs should check out my attempt at this in the Nomad repo.

On Nov 19, 2014, at 7:04 PM, Bob Carpenter notifications@github.com wrote:

To remove duplication from our derivative testing code and also make it portable for when we upgrade autodiff, we need to move our testing into a generic functional framework that takes pairs of functors and compares them.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub.

[syclik]

This issue was moved to stan-dev/math#51

[syclik]

This issue was moved to stan-dev/math#55