Add sanity checks for multi-level instancing

[ingowald]

Current API allows user to specify max instancing depth, and then sets optix up in the right way ... but does not actually check whether the created graph of group nodes adheres to the specified limit.

Should add a check at least upon group creation time that tracks, for each created group, how deep the tree below it is, and if that matches the context's configured depth.

[ingowald]

fixed