Possible bugs

[hailuu684]

Hello,
I am testing your repo and I got this error. I am not sure if this is an issue in different torch versions or incompatibilities.

I got an error in d.grad.add_(g). It says 'NoneType' object has no attribute 'add_'. I printed the type of 'd', it is obviously a Tensor type. I do not understand why can cause this problem.

[hailuu684]

May I ask a question? I am curious about if I can use the generated distilled dataset that you already published to train with normal procedure. What I meant by the normal procedure is like the image below. Or do I have to write the customized network as you did?

[ssnl]

You may use the distilled dataset however you want. Whether to reparametrize the network shouldn't really affect results. However, other changes in training (e.g., network architectures, learning rates, epochs, etc.) can affect the results non-trivially.

[ssnl]

Re your original question: that shouldn't happen if you are running the code as-is, since there is this line

dataset-distillation/train_distilled_image.py

Line 70 in 1c9657a

p.grad = torch.zeros_like(p)

[hailuu684]

Thank you for your quick reply, I haven't modified the code but still got this error. Can you suggest where should I take a look on? Honestly, your code is advanced to me.

[ssnl]

It might be a torch version issue then because I didn't see this before. I guess you can try to change code to test

• if d.grad is None, just g assign to d.grad
• otherwise, add g to it.

Although I haven't experienced the error you are seeing so I can't guarantee correctness.

[hailuu684]

Thank you for the instruction, here is the updated code for anyone who is having the same error like me

[hailuu684]

May I ask this question? In your code, are you using real data to generate distilled dataset? and then training on the distilled dataset? However, in the function init_data_optim, I see distill_label and distill_data are generating randomly. I understand that this is the initialization, but in which part of your code updates the distilled data? would you mind if you could point it out for me? Thank you very much.

[ssnl]

@hailuu684 In this code, distilled data are initialized as random noise. But yes, many works show that initializing from real images can work better in certain cases. It is not implemented here.