Adding exposure control for Kinect2

[saulthu]

Has anyone investigated adding manual exposure control to libfreenect2?

I know it is possible to manually control exposure of the Kinect2 by using a library that Microsoft included in a project they released on GitHub: https://github.com/Microsoft/MixedRealityCompanionKit/tree/master/KinectIPD/NuiSensor

This library, "NuiSensorLib", is a precompiled binary for Windows. I currently use it with a small C++ wrapper to set the frame integration time and the analog gain of the Kinect2 during recording. The relevant library function is NuiSensor_ColorChangeCameraSettings, and the other files beside this library file have examples of usage.

I would like this functionality within libfreenect2 so I can control multiple Kinect2 devices on a single PC.

The code I use constructs a command message with a sequence number of zero, and an RGB setting command integer and a value integer. The problem is that I suspect the NuiSensor_ColorChangeCameraSettings function then wraps this in another command structure before sending over USB, and since it is a pre-compiled library, I don't have the values that it is using.

Has anyone else looked into this library already? Or have any hints on where I could look next?

Thanks 😄

[floe]

Very interesting, thanks for pointing this out. Can you post a link to your wrapper library? IIRC this sounds quite similar to how the raw USB commands for the Kinect are constructed, so maybe it's actually quite straightforward to implement this in libfreenect2. Just to be sure, this only applies to the color camera, correct?

[saulthu]

Yes, this is only about the color camera settings (integration time in milliseconds, analog gain, white balance settings, etc.) and NuiSensor.h contains additional color camera settings that can be read/written.

I've uploaded my C++ wrapper (and conversion from C++/CLI) at https://github.com/saulthu/Kinect2Exposure

I agree, this looks a lot like the USB commands within libfreenect2. The code within this wrapper boils down to something roughly like this (but this does leave out some initialization):

NUISENSOR_HANDLE sensor;
NuiSensor_Initialize(&sensor)
// ... NUISENSOR_RGB_COMMAND_SET_ACS command is sent once after connection ...

// Example exposure command.
const uint32_t command = NUISENSOR_RGB_COMMAND_SET_INTEGRATION_TIME;
const uint32_t argument = _float_to_command_int(20.0f)

NUISENSOR_RGB_CHANGE_STREAM_SETTING settings;
settings.NumCommands = 1;
settings.SequenceId = 0;
settings.Commands.resize(1);
settings.Commands[0].Cmd = command;
settings.Commands[0].Arg = argument;

std::vector<BYTE> scratchBuffer(1024);
NUISENSOR_RGB_CHANGE_STREAM_SETTING_REPLY reply;
int reply_size = // macro to compute reply size
NuiSensor_ColorChangeCameraSettings(
	sensor,
	scratchBuffer, scratchBuffer.size(),
	&settings, settings_size,
	reply, reply_size);

I expect the full command packet will need some prefix bytes to indicate that this is a "color change camera settings" command, since the RGB commands settable via this library interface are values for use within an RGB context -- the NUISENSOR_RGB_COMMAND_* commands range [0, 83], which looks to me like we need to wrap this so Kinect knows this is not a register value, or a status read, etc. e.g. if we were to use CommandWithNParam<> we'd need to know what CommandId to use, and I guess for this NuiSensorLib the command ID is hidden within the NuiSensor_ColorChangeCameraSettings call.

On a side note (not the goal of the discussion here), I suspect this NuiSensorLib.lib library is what Microsoft uses to implement connection to Kinect2 devices within KinectService.exe. This library allows enumerating and opening of multiple Kinect2 devices without the KinectService.exe running. I have managed to capture raw IR frames but I haven't succeeded at starting the color stream.

[saulthu]

Using Visual Studio to step into the assembly of NuiSensorLib.lib I see that the function NuiSensor_ColorChangeCameraSettings calls into a function:

NuiSensorGenericSendAndReceiveAsync(
    _NUISENSOR_HANDLE * NuiSensorHandle,
    void * ScratchBuffer,
    unsigned long ScratchBufferSize,
    unsigned long Command,
    unsigned char * CommandBytes,
    unsigned long CommandSize,
    unsigned char * SendBuffer,
    unsigned long SendBufferSize,
    unsigned char * ReceiveBuffer,
    unsigned long ReceiveBufferSize,
    unsigned long * BytesTransferred,
    _OVERLAPPED * pOverlapped)

and the passed value of Command is 62 (base 10), CommandSize is 16, and an internal variable commandPacketSize is 36.

[saulthu]

Further, interpreting the ScratchBuffer as an array of 9 32-bit integers (which matches the commandPacketSize of 36 bytes) gives the following:

	Name	Value	Type
◢	(unsigned int*)ScratchBuffer, 9	0x000001da1f91c7a0 {100802569, 0, 16, 62, 0, 1, 0, 21, 1065353216}	unsigned int[9]
	[0]	100802569	unsigned int
	[1]	0	unsigned int
	[2]	16	unsigned int
	[3]	62	unsigned int
	[4]	0	unsigned int
	[5]	1	unsigned int
	[6]	0	unsigned int
	[7]	21	unsigned int
	[8]	1065353216	unsigned int

Of note, the [3] = 62 matches the Command passed into NuiSensorGenericSendAndReceiveAsync, and the final two values 21 is the command I was sending at the time (NUISENSOR_RGB_COMMAND_SET_ANALOG_GAIN) and the 1065353216 value is the gain value being set, which is float(1.0). I'm not entirely sure what all the values are, but from what I can tell:

• [0] = 100802569 = 0x6022009 is assigned to the buffer as a constant in the assembly
• [1] assuming this is a packet sequence number
• [2] = 16 assuming this is the response length -- it seems to be the same value for a variety of RGB commands when setting or getting a value
• [3] = 62 is the Command value which is unconditionally set to the buffer after the constant header
• [4] assuming this is an unused value
• [5 ... 8] is memcpyed from the contents of the NUISENSOR_RGB_CHANGE_STREAM_SETTING structure
  • [5] = 1 means NumCommands = 1
  • [6] = 0 means SequenceId = 0
  • [7] = 21 means Cmd = 21 = NUISENSOR_RGB_COMMAND_SET_ANALOG_GAIN
  • [8] = 1065353216 means Arg = float(1.0) as uint32_t, which is the gain value being set in this example

[saulthu]

It appears that the call made to NuiSensorGenericSendAndReceiveAsync has CommandBytes passed into it as the pointer to the NUISENSOR_RGB_CHANGE_STREAM_SETTING structure, which makes sense that in this example the CommandSize is 16 (4 uint32_t values: NumCommands, SequenceId, Cmd, Arg), so everything in the completed packet from index [5]-onward is simply a copy of that structure.

[saulthu]

The same packet structure is used to get a value back:

	Name	Value	Type
◢	(unsigned int*)ScratchBuffer,9	0x00000290490fc7a0 {100802569, 0, 16, 62, 0, 1, 0, 26, 0}	unsigned int[9]
	[0]	100802569	unsigned int
	[1]	0	unsigned int
	[2]	16	unsigned int
	[3]	62	unsigned int
	[4]	0	unsigned int
	[5]	1	unsigned int
	[6]	0	unsigned int
	[7]	26	unsigned int
	[8]	0	unsigned int

This is a packet with the Cmd = 26 = NUISENSOR_RGB_COMMAND_GET_ACS, and the Arg is always set to zero for getting values.

[saulthu]

From looking at libfreenect2/include/internal/libfreenect2/protocol/command.h it appears we can do this quite simply as you suggested. If we define a new command e.g.

#define KCMD_RGB_SETTING 0x3E  // Command value for color camera settings

and then a struct for the RGB command, something like the following (but I don't know what exactly the reply should be)

// FIXME: set response length to correct value -- assuming 16 to match observed packet
struct SingleColorSettingCommand : public Command<KCMD_RGB_SETTING, 16, 16, 4>
{
  SingleColorSettingCommand(uint32_t cmd, uint32_t value)
  : Command<KCMD_RGB_SETTING, 16, 16, 4>(0)  // seq always zero
  {
    // Data parameters are elements of struct NUISENSOR_RGB_CHANGE_STREAM_SETTING
    // which supports multiple settings for a single call.
    this->data_.parameters[0] = 1;  // NumCommands = 1 for single command
    this->data_.parameters[1] = 0;  // SequenceId = 0 for RGB commands
    this->data_.parameters[2] = cmd;  // Cmd with a value from NUISENSOR_RGB_COMMAND_*
    this->data_.parameters[3] = value;  // Arg is int or float depending on Cmd
  }
  
  // Could overload ctor to ease usage for float-valued settings.
  SingleColorSettingCommand(uint32_t cmd, float value)
  : SingleColorSettingCommand(cmd, _float_bytes_to_uint32(value))
  {
  }
};

[saulthu]

OK, I'll make a pull-request once I've made a working example.

As another aside, the header NuiSensorLib.h contains a lot of information about data structures that clears up many previous unknowns e.g. the calibration data structures can be fully seen in this header e.g. DepthCameraParamsResponse.unknown0 is actually NUISENSOR_CALIBRATION_TABLE_BLOCK.AlphaC which corresponds to the calibration Skew coefficient.

[floe]

Thanks a lot, I will definitely have a look at the calibration parameters, that might help to clear up some missing bits and pieces all over libfreenect2.