questions about decl_buffer

[xqdan]

Hi all,

I noticed that in case test_schedule_tensorize.py, when modify offset_factor=16, to 7, a assert would come out,

assert((((j.outer*16384) % 7) == 0), "Argument W.elem_offset has an unsatisfied constraint")

In tutorial, it's said that, elem_offset is required to be multiple of offset_factor. looks like it's checking for this constraint here.
But why we need this constraint ?
What's the decl_buffer lower to , especially for the shape like (m,n) ?
Any differences between decl_buffer, vload, vstore, cache_read, cache_wirte, and double_buffer?

Thanks!

[tqchen]

Please see the document here http://docs.tvmlang.org/api/python/tvm.html#tvm.decl_buffer

If you have further questions, it is great, and we would be happy to discuss and answer them. It would be even better if you can open a PR to update the documents that improve the understanding so the future users will benefit from the discussion

[xqdan]

@tqchen sure, I'd love to contribute some to nnvm/tvm.
I've seen this link, still confused with questions above, could you explain a little bit?

seems that decl_buffer is like a special tensor placeholder, and elem_offset, offset_factor, stride together with other things are attributes of this placeholder. when we do some transformations like tile, split with a transform-factor, tvm will do some checking according to the attributes of decl_buffer. what i don't understand is why check with constraint elem_offset%offset_factor==0? To me, elem_offset is calculated based on offset_factor or transform-factor. similar with iteration variable based on loop index, but loop index is continuous. If we unroll a loop with unroll_factor, we always check if there is a remainder loop, however in tensor transformation, there is no remainder, if there is, should be padded before transforming. it would be clear if we have a formula or something.

the other thing I'm thinking is how to lower this special placeholder to intrinsics of different targets, like tensor load/store, and tensor multiply/add.

the last thing is I don't have a whole picture in my mind about these concepts, decl_buffer, vload, vstore, cache_read, cache_wirte, and double_buffer.

thanks!

[tqchen]

Buffer is a protocol of how data can be layout out in DL Tensor in the function arguments when we make API. For example, if the strides exist, then the data can contain strides, this kernel will be quite different from a kernel for compact tensor. Similar things like elem_factor give hint about the alignment of the data and can let the compiler know whether vectorization can be done or not.

These are advanced features and usually they comes with default, for example by default TVM build kernel that only accepts DLTensor with elem_offset=0 and no strides field

[xqdan]

if the strides exists, say stride=1, the op will access tensor data across stride=1, or stride*offset_factor? I mean, is there any connection between stride and offset_factor? besides, you mentioned compact tensor, so for tensor with stride, is it sparse tensor? And for compiler, do you mean nnvm compiler?

[tqchen]

strides is used to mark the stride on each tensor dimension so that each dimension do not have to be strictly continuous. see https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.ndarray.strides.html

The offset_factor has things to do elem_offset in DLPack.

[xqdan]

Ok, I'm trying to explain it with examples below, please correct it if wrong,

as the example below,
elem_offset = sum(y.strides * np.array((1,1,1))),

which indcates the address offset to np.array((0,0,0) in bytes.

y.strides is a vector in this case, and np.array((1,1,1) is 17, the index of one dimension, in this way(single dimension) load/store data in physical memory.

so we can access data via elem_offset, also we can do split or vectorize according to elem_offset.

> >>> y = np.reshape(np.arange(2*3*4), (2,3,4))
> >>> y
> array([[[ 0,  1,  2,  3],
>         [ 4,  5,  6,  7],
>         [ 8,  9, 10, 11]],
>        [[12, 13, 14, 15],
>         [16, 17, 18, 19],
>         [20, 21, 22, 23]]])
> >>> y.strides
> (48, 16, 4)
> >>> y[1,1,1]
> 17
> >>> offset=sum(y.strides * np.array((1,1,1)))
> >>> offset/y.itemsize
> 17
> >>> y.itemsize
> 4

for the case as below in TVM, strides is not defined in bytes, offset_factor is the same meaning as y.itemsize above. in TVM strides*offset_factor is the same meaning as y.strides above.
strides=[tvm.var('ldw')*offset_factor, 1*offset_factor]

Wb = tvm.decl_buffer(w.shape, w.dtype,
                         name="W",
                         offset_factor=16,
                         strides=[tvm.var('ldw'), 1])

[tqchen]

The index offset equals

elem_offset + sum(index[i] * strides[i] for i in range(ndim))

elem_offset is the offset to the head address. The reason that this need to be independent field is because this helps us to deduce alignment. (The head pointer can be opaque and always aligned)

[tqchen]

Hope my last answer makes the question clear and I am going to close this for now. You are more than welcomed to update the docstring to incoporate your understanding and help others in the community:)

[tigert1998]

I am trying to work out what offset_factor means. I think the basic idea is right. But in detail, there're mistakes. According to xqdan, offset_factor has the same meaning with y.itemsize. But I think not. In TVM's official tutorial (https://tvm.apache.org/docs/how_to/work_with_schedules/tensorize.html), offset_factor is 1 in intrin_gemv. But at the same time, a, b and k all has float32 type. So I suspect offset_factor means the interval (counted by the number of elements rather bytes) between two nearby elements.