Add op rrelu #9736
Add op rrelu #9736
Conversation
youxiudeshouyeren
requested review from
hjchen2,
BBuf,
jackalcooper,
daquexian and
liujuncheng
as code owners
youxiudeshouyeren
requested review from
oneflow-ci-bot
and removed request for
oneflow-ci-bot
| test_case.assertTrue( | ||
| np.allclose( | ||
| (flow.where(flow_tensor >= 0, 1, 0)).cpu().detach().numpy(), | ||
| (flow.where(flow_div == 1.0, 1, 0)).cpu().detach().numpy(), | ||
| ) | ||
| ) | ||
| test_case.assertTrue( | ||
| np.allclose( | ||
| (flow.where(flow_tensor < 0, 1, 0)).cpu().detach().numpy(), | ||
| ( | ||
| flow.where( | ||
| flow.logical_and( | ||
| flow.logical_and(flow_div >= rate, flow_div <= (rate + 0.5)), | ||
| flow_tensor < 0, | ||
| ), | ||
| 1, | ||
| 0, | ||
| ) | ||
| ) | ||
| .cpu() | ||
| .detach() | ||
| .numpy(), | ||
| ) | ||
| ) |
There was a problem hiding this comment.
这里测试的是原来大于等于0的位置，还是原数据；小于0的位置，元素乘的因子在输入参数的区间内部。
| OF_LAUNCH_BOUNDS_2(block_size_bound, grid_size_bound) | ||
| __global__ | ||
| void RReluKernel(int64_t numel, uint64_t seed, uint64_t offset, const T* in_ptr, T* out_ptr, | ||
| T* noise_data_ptr, Distribution dist_func, Transform transform_func) { |
There was a problem hiding this comment.
这里没有必要再实现一遍，可以使用DistributionElementwiseGridStrideKernel模板函数，二者实现一样的
There was a problem hiding this comment.
RReLU需要传递三个指针进去，并且还存在判断，DistributionElementwiseGridStrideKernel好像不能实现。单修改transform_func的话，也需要修改DistributionElementwiseGridStrideKernel中transform_func那部分的接口，因为DistributionElementwiseGridStrideKernel只是赋值了随机数，没有判断大小和乘法的那部分。
There was a problem hiding this comment.
这里没有必要再实现一遍，可以使用
DistributionElementwiseGridStrideKernel模板函数，二者实现一样的
DistributionElementwiseGridStrideKernel在运行时只给transform_func传递了一个random_val的随机数，我这里需要input进行条件判断，out指针来进行和随机数的乘法，没办法直接用DistributionElementwiseGridStrideKernel模板函数
There was a problem hiding this comment.
这里没有必要再实现一遍，可以使用
DistributionElementwiseGridStrideKernel模板函数，二者实现一样的
DistributionElementwiseGridStrideKernel在运行时只给transform_func传递了一个random_val的随机数，我这里需要input进行条件判断，out指针来进行和随机数的乘法，没办法直接用DistributionElementwiseGridStrideKernel模板函数
明白了，这里有两个输出
对接pytorch的这个算子： https://pytorch.org/docs/stable/generated/torch.nn.RReLU.html?highlight=rrelu#torch.nn.RReLU