Something is wrong on some of the output images?

[cfanyyx]

I used the default settings and params. After training on my own datasets, in some of my test output images using the saved model, there are strange green areas on them like the image on the right below(the left one is the origin test image). Is this related to the Bilateral Learning algorithm?

[mgharbi]

Looks like some dimensions are transposed, are you having the same issue with the pretrained models?

[cfanyyx]

@mgharbi Thanks for your reply. Yes, it is! Here are some of the results below. The left one is processed by pretrained model of hdrp, and the right one is processed by pretrained model of instagram. I think using other pretrained models will also get the similar issue.

[mgharbi]

Ah crap, if your TF might have changed some the internal storage pattern of the tensors somewhere, this is definitely not the expected output. The provided sample image sample_data/input.png should work, if not we'll have to dig deeper.

One way to debug this would be to isolate the "bilateral_slice_apply" operator and feed it with simple images e.g. an image with R=1 G=2 B=3, and a bilateral grid that contains all zeros except for one of the 12 coefficients. For instance with coef0=1 only, the output should be coef0R + coef1G +coef2B + coef3 = 11 = 1 everywhere, if that is not the case something got transposed

[cfanyyx]

@mgharbi Only partial test output results are bad and the others are good. There is no problem on the output of the sample image. In addition, I found that the guidance map of the input image was already corrupt.

[cortes93]

@mgharbi @cfanyyx Hi, i have the same issue with just some images, depending on their shape.
The strange thing is that if you substitute these lines

hdrnet_legacy/hdrnet/models.py

Lines 177 to 185 in b06d011

	guidemap = tf.contrib.layers.convolution2d(
	inputs=guidemap,
	num_outputs=1, kernel_size=1,
	weights_initializer=tf.constant_initializer(1.0/nchans),
	biases_initializer=tf.constant_initializer(0),
	activation_fn=None,
	variables_collections={'weights':[tf.GraphKeys.WEIGHTS], 'biases':[tf.GraphKeys.BIASES]},
	outputs_collections=[tf.GraphKeys.ACTIVATIONS],
	scope='channel_mixing')

which learn to convert an RGB image in a grayscale one, with guidemap = tf.image.rgb_to_grayscale(curve) the problem disappears.
I still don't know why but maybe @mgharbi has some ideas about it.

[mgharbi]

This fix surely would guide you a guide, but you will not be able to learn a better occupation of the grid (via the tonecurve).

I would also be surprised if the conv2d layer was bogus, and I suspect the logic that comes before that to have turned stale:

hdrnet_legacy/hdrnet/models.py

Lines 154 to 173 in b06d011

	with tf.name_scope('ccm'):
	ccm_bias = tf.get_variable('ccm_bias', shape=[nchans,], dtype=tf.float32, initializer=tf.constant_initializer(0.0))
	guidemap = tf.matmul(tf.reshape(input_tensor, [-1, nchans]), ccm)
	guidemap = tf.nn.bias_add(guidemap, ccm_bias, name='ccm_bias_add')
	guidemap = tf.reshape(guidemap, tf.shape(input_tensor))
	# Per-channel curve
	with tf.name_scope('curve'):
	shifts_ = np.linspace(0, 1, npts, endpoint=False, dtype=np.float32)
	shifts_ = shifts_[np.newaxis, np.newaxis, np.newaxis, :]
	shifts_ = np.tile(shifts_, (1, 1, nchans, 1))
	guidemap = tf.expand_dims(guidemap, 4)
	shifts = tf.get_variable('shifts', dtype=tf.float32, initializer=shifts_)
	slopes_ = np.zeros([1, 1, 1, nchans, npts], dtype=np.float32)
	slopes_[:, :, :, :, 0] = 1.0
	slopes = tf.get_variable('slopes', dtype=tf.float32, initializer=slopes_)

instead of rgb_to_grayscale, you could try to set inputs=input_tensor in L177 to verify this part.

If the output looks like an image (no wieird transpose), the next steps would be:

1. isolate the curve part: you could comment out L163-175 and sum guidemap along the channel dimension to get a valid grayscale image.
2. isolate the change of color basis ccm (for color correction matrix): comment L154-160. The dimension of guidemap should still be valid to use as argument to the tonecurve below.

[cfanyyx]

@mgharbi @cortes93 After testing, it seems that there is something wrong with ccm, the part of curve looks normal, and the conv2d also has something wrong, but the problem of conv2d may come from its prior ccm as it is a convolution layer.

[cortes93]

@mgharbi @cfanyyx I think I found a workaround for the issue which let us maintain the convolutional layer. You need to run the multiplication in L157 and the convolution in L177 with the CPU. I don't know the reason but I think that it is due to a mismatch between the way that GPU learned to multiply the two matrix and the shape of the input we want to make inference.