Mutable arrays for intermediary variables

[JWHennessey]

Hi, really great work - thanks!

I would like to have a mutable Array (OptImage) that is dependent on some unknown parameters and can easily be read from the GPU.

Specifically I have some Unknowns which are used to generate intermediary arrays which are used in my fitting term. Some pseudo code would be:

--Input
local I                 = Array("d",  opt_float3,{W,H},6)       -- Target Image

--Unknowns 
local ps                = Unknown("ps", opt_float,{W,H},3)      -- scale factor
local n                 = Unknown("n",  opt_float,{W,H},4)      -- specular exponent
local A                 = Unknown("a",  opt_float3,{W,H},5)     -- albedo 

--Dependent On Unknowns (Zero when initialized)
local D                 = Array("d",  opt_float3,{W,H},6)       -- Diffuse Map 
local S                 = Array("ss", opt_float3,{W,H},7)       -- Specular Shading
local S_d               = Array("ds", opt_float3,{W,H},8)       -- Diffuse Shading

function specularShading(albedo, spec_n, spec_e, normal, light)
-- Some example rendering function
end
.....

S(0,0) = specularShading(A(0,0), n(0,0), ps(0,0))

....
....

Energy(A(0,0) * S_d(0,0) + S(0,0) - I(0,0))

Currently I get a compile time error when assigning a value to an Array e.g S(0,0) = 1.0. So instead I just make S local e.g local S = 1.0 but then cannot read the resulting intermediary images to the CPU. While I can implement specularShading on the CPU using the output unknown, this increases my running time and for debugging it would be much easier to view arbitrary intermediary variables.

Is this currently possible another way or is this just a feature request?

Thanks!

[Mx7f]

I'm glad you are finding Opt useful!

The closest feature to what you are requesting that we have in Opt right now is the ComputedArray() functionality (see the end of section 3 in https://arxiv.org/pdf/1604.06525.pdf or the ShapeFromShading example), which is used to balance recompute and bandwidth. As a side effect it allocates intermediary GPU arrays for the computed value (and its partial derivatives), but we currently have no API for accessing said values outside of Opt.

I could see adding a "void Opt_GetSolverParameter(Opt_State* state, Opt_Plan* plan, const char* name, void** valueHandle);" or something similar which allowed access to said computed arrays, though that is further baking in constraints on the solver into the API in this early stage of development. I will think through this later this week.

Another option, of course, would be to generally allow array to be read/write like you propose. This seems interesting, but would definitely require some reworking in the backend. I'll give some thought to see if there's a clean way of handling it.

A halfway measure you could do right now to avoid most of the running time overhead from computing on the CPU is to just implement a simple CUDA kernel using the unknown array. This requires keeping the energy function and the kernel in sync (less than ideal and something we should solve in the future), but the overhead should be minimal, as you only need to run it once per nonlinear iteration.