add method to make tensor constant for debug purposes

[noskill]

🚀 Feature

Add method to make constant tensors

tensor.make_const()

Motivation

This option will make it easy to find inplace operators in case of such exceptions:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

Instead RuntimeError should be thrown when inplace operation is performed on constant tensor.

Alternatives

Perhaps it is even better to make all tensors constant by default, since inplace operators rarely needed.

cc @ezyang @ssnl @albanD @zou3519 @gqchen

[zou3519]

https://pytorch.org/docs/stable/autograd.html#torch.autograd.detect_anomaly might solve the problem. IIRC it should tell you where in the forward pass you used an op that caused an error in backwards.

[albanD]

I would agree that this would be a useful feature to have in general.
For this particular use case, you can indeed, as @zou3519 mentioned, use anomaly mode to pinpoint the error.

[zou3519]

One thing we could take design inspiration from is numpy's flags: https://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.flags.html. One can set a numpy array's WRITEABLE flag to off, making the array read-only.

[noskill]

Yes, i am trying to pinpoint the place with detect_anomaly and doing tensor.clone(), so making all the data readonly would be really helpful

[ezyang]

We've talked about this internally before too, it's a generally good idea we should support.

[noskill]

Where should check for the flag be placed? I am not familiar with the source yet..

[ezyang]

Not clear; the design for this may be fairly involved :)

[t-vi]

Maybe one could make this more general: we could introduce a hook if a tensor is modified inplace. This is a bit complicated as we would make it "if the storage underlying the tensor is modified inplace", but I think it would have more general applications, too (e.g. caching computations which only need to be re-computed when inputs change).