Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix saliency map bugs #886

Merged
merged 1 commit into from
Jun 24, 2021
Merged

fix saliency map bugs #886

merged 1 commit into from
Jun 24, 2021

Conversation

BillyCheung10botics
Copy link
Contributor

This PR fixed the following two bugs:

  • tf2 version incomplatibility
    First, the codes constructing saliency models were incompatible with the tensorflow version 2.

    modifier_fn = get('guided')
    sal_model_mod = modifier_fn(sal_model)
    losses = [
    (ActivationMaximization(sal_model_mod.layers[layer_idx], None), -1)
    ]
    self.opt = Optimizer(sal_model_mod.input, losses, norm_grads=False)
    return True
    def compute_visualisation_mask(self, img):
    grad_modifier = 'absolute'
    grads = self.opt.minimize(seed_input=img, max_iter=1, grad_modifier=grad_modifier, verbose=False)[1]

    We should use the GradientTape() of tensorflow to calculate the gradient.

  • uint8 conversion bug
    Second, the conversion of RGB values from float (0.0, 1.0) to int (0, 255) was wrong,

    image = image * 255
    image = image.astype('uint8')

    which creates artifacts as the bottom left figure shown. The red/yellow/blue parts are artifacts with RGB values over 255. Since some float values of the image array can be great than 1.0. I use cv2.normalize to properly convert the RGB values from float to uint. The correct saliency map is represented as the yello-purple clould as the right figure shown.
    wrong RGB conversion
    (left): video with artifacts (right): video with correct saliency mask

- tf2 version incomplatibility
- uint8 conversion bug
Copy link
Contributor

@DocGarbanzo DocGarbanzo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, but I'm wondering if the Gaussian Blur is the source of having the salient mask showing much less localised impact than in the original algorithm. Could you make this switchable, so the user could choose between the 'old' version of overlay vs the new one?
It seems you have done this completely without using the keras-viz package, that is great. From what I recall that was using a single step of backprop to utilise a single gradient calculation, but it seems this can be done much better with using the keras gradient (or GradientTape in tf 2.0) directly between the input and output of the salient-adjusted model. Can you confirm?

donkeycar/management/makemovie.py Show resolved Hide resolved
donkeycar/management/makemovie.py Show resolved Hide resolved
donkeycar/management/makemovie.py Show resolved Hide resolved
Copy link
Contributor

@DocGarbanzo DocGarbanzo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great change, many thanks.

@DocGarbanzo DocGarbanzo merged commit 8a1562e into autorope:dev Jun 24, 2021
@BillyCheung10botics BillyCheung10botics deleted the fixsaliency branch July 21, 2021 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants