Skip to content

Commit

Permalink
Use flow.randint in dataloader (#6086)
Browse files Browse the repository at this point in the history 
* add randint

* add

* add doc test

* Update randint_kernel.cu

* Update randint_kernel.cpp

* Update randint_op.cpp

* reconstrcut

* refine the code

* add test

* add test

* add test

* format

* Dev randint refine (#5981)

* disable backward pass consistent tensor meta check. (#5871)

* disable backward pass consistent tensor meta check.

* auto format by CI

Co-authored-by: binbinHan <han_binbin@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* ddp broadcast params and buffers (#5913)

* ddp broadcast params and buffers

Signed-off-by: daquexian <daquexian566@gmail.com>

* auto format by CI

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* add clang tidy target (#5957)

* add clang tidy target

* fix a bug

* refine

* refine

* reformat

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* cfg: add move assignment operator for performance (#5962)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add zhangshen op-test (#5600)

* add some op-test

* fix dims_error in my branch

* Fix the bad backward kernel function by using 'cuda::atomic::Add' (#5614)

* Test `nn.AdaptiveAvgPoolXd` (#5615)

* Fix the bad backward kernel function by using 'cuda::atomic::Add'

* Support the 'NoneType' annotation

* Support objects of 'collections.abc.Iterable' as 'output_size'

* Test with all cases of 'output_size'

* Update adaptive_pool_gpu_kernel.cu

* Skip testing `nn.AdaptiveAvgPool3d` for the current PyTorch

* remove some useless test

* Format TODO

* Add the assertion messages for 'output_size'

* Reformat codes

* Remove raw tests for `flow.negative`

* Remove unnecessary codes and add the assertion messages

* Merge updates for 'generators.py' from master

* Remove unnecessary 'random()'

* Delete the separate test for `AvgPool2d`

* Fix import paths

* Fix import problems

* Remove the PyTorch import

* Denote the annotations for `tile` and `repeat` ops

* Add the test for `nn.AvgPool1d`

* Choose better generators for `nn.MaxPoolXd`

* Randomly choose `dilation` and default values

* auto format by CI

* Test more kwargs for `nn.AvgPoolXd`

* Add tests for `return_indices`

* auto format by CI

Co-authored-by: Tianyu Zhao <guikarist@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* fix wrong names (#5951)

* fix wrong names

* auto format by CI

* refine

* auto format by CI

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Enable more checkers for clang-tidy in CI (#5738)

* CI: enable more checkers for clang-tidy

* .clang-tidy: remove cppcoreguidelines-pro-type-vararg

* CI: remove duplicate checkers

* CI: remove clang-analyzer-alpha.deadcode.*

* .clang-tidy: add performance-*

* oneflow/core/eager: remove unnecessary malloc & free

* .clang-tidy: add clang-analyzer-cplusplus.* to werror

* user_kernel: remove useless move

* quantization_aware_training: fix move return

* .clang-tidy: add google-*

* CI: fix clang tidy command

* CI: fix test

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Feat grad mode classes (#5956)

* feat(no_grad): support no_grad decorator

* feat(AutogradMode): export flow.autograd_mode

* feat(GradMode): export some grad_mode class

* docs(GradMode): export documents

* refine

* docs(GradMode): export document for is_grad_enabled

* auto format by CI

* fix(GradMode): fix single client bug

* fix bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* extract_consistent_to_consistent_op_expr (#5870)

* abstract_consistent_to_consistent_op_expr

* fix compiler complaint

* refactor consistent-to-consistent eager consisitent op interpreter

* fix compiler complaint

* refactor ConsistentToConsistentOpExpr

* lazy interpreter (#5903)

* fix bugs about consistent_id

* refactor functional::ToConsistent

* refactor GetNdSbp

* Update eager_consistent_op_interpreter.cpp

* Update eager_mirrored_op_interpreter.cpp

* fix error

* fix error

* auto format by CI

* Update nd_sbp.h

* refine identity boxing

* fix sync checkmeta error

* avoid consistent id check in lazy

Co-authored-by: Xinqi Li <lixinqi0703106@163.com>
Co-authored-by: leaves-zwx <kunta0932@gmail.com>
Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* add CMAKE_INTERPROCEDURAL_OPTIMIZATION in fast cmake cache (#5970)

* add CMAKE_INTERPROCEDURAL_OPTIMIZATION in fast cmake cache

* skip test targets of re2

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* check: fix clang-tidy-diff commands (#5972)

* check: fix clang-tidy-diff commands

* CI: fix step names

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Cpu mpi (#5865)

* cuda base cpu mpi boxing

* cpu_mpi

* fix conflicts

* add cpu mpi unittests

* more checks and unittests

* abstract_consistent_to_consistent_op_expr

* fix compiler complaint

* refactor consistent-to-consistent eager consisitent op interpreter

* fix compiler complaint

* refactor ConsistentToConsistentOpExpr

* lazy interpreter (#5903)

* fix bugs about consistent_id

* more test_consistent_cast unittests

* refactor functional::ToConsistent

* refactor GetNdSbp

* fix compiler complaints

* refactor GetDevice4CurrentProcessCtx

* fix error

Co-authored-by: clackhan <han_binbin@163.com>
Co-authored-by: leaves-zwx <kunta0932@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix_bug_test_tensor_str (#5958)

* fix bug int test_tensor_str

* format

* fix comment

* fix bug to(cuda) is unavailable in cpu env

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* common/error: fix build error in mac (#5971)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Prevent running oneflow in forked subprocess (#5976)

* prevent_running_oneflow_in_forked_subprocess

* add line change

* IsFork => IsForkedSubProcess

* auto format by CI

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* refine randint

Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: binbinHan <han_binbin@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: Peihong Liu <mosout@qq.com>
Co-authored-by: Twice <i@twice.moe>
Co-authored-by: ZhangShen <55383772+zhangshen12356@users.noreply.github.com>
Co-authored-by: Tianyu Zhao <guikarist@gmail.com>
Co-authored-by: Luyang <flowingsun007@163.com>
Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>
Co-authored-by: Xinqi Li <lixinqi0703106@163.com>
Co-authored-by: leaves-zwx <kunta0932@gmail.com>
Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com>
Co-authored-by: liufengwei0103 <2472937968@qq.com>

* refine

* refine

* auto format by CI

* refine

* Update functional_api.yaml

* Update functional_api.yaml

* refine the code

* auto format by CI

* refine

* fix ci error

* replace np randint with flow.randint

* add tensor.item tensor.tolist

* add test case

* refine

* auto format by CI

* fix ci fail

* revert change

* auto format by CI

* refine

* auto format by CI

Co-authored-by: Kevin-XiongC <kevin_xiong1997@outlook.com>
Co-authored-by: Bowen Chen <bob2420083992@gmail.com>
Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: binbinHan <han_binbin@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: Peihong Liu <mosout@qq.com>
Co-authored-by: Twice <i@twice.moe>
Co-authored-by: ZhangShen <55383772+zhangshen12356@users.noreply.github.com>
Co-authored-by: Tianyu Zhao <guikarist@gmail.com>
Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>
Co-authored-by: Xinqi Li <lixinqi0703106@163.com>
Co-authored-by: leaves-zwx <kunta0932@gmail.com>
Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com>
Co-authored-by: liufengwei0103 <2472937968@qq.com>
  • Loading branch information
@Flowingsun007 @Kevin-XiongC
@VertexC @lixinqi @clackhan @oneflow-ci-bot @daquexian @mosout @PragmaTwice @zhangshen12356 @wyg1997 @leaves-zwx @jackalcooper @liufengwei0103
17 people committed Aug 30, 2021
1 parent 2625e83 commit d23f3a1
Show file tree
Hide file tree
Showing 2 changed files with 16 additions and 36 deletions.
34 changes: 12 additions & 22 deletions python/oneflow/utils/data/sampler.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -141,34 +141,24 @@ def __iter__(self):
n = len(self.data_source)
if self.generator is None:
generator = flow.Generator()
generator.manual_seed(
# TODO: replace with flow.randint
flow.Tensor(1, dtype=flow.int64)
.uniform_(0, 999)
.numpy()
.item()
)
generator.manual_seed(np.random.randint(0, np.iinfo(np.int64).max))
# TODO: use Tensor.random_
# generator.manual_seed(
# int(flow.empty((), dtype=flow.int64).random_().item())
# )
else:
generator = self.generator
if self.replacement:
np.random.randint()
for _ in range(self.num_samples // 32):
yield from np.random.randint(
high=n, size=(32,), dtype=np.int64
yield from flow.randint(
high=n, size=(32,), dtype=flow.int64, generator=generator
).tolist()
# TODO: use flow.randint replace np.randint
# yield from flow.randint(
# high=n, size=(32,), dtype=flow.int64, generator=generator
# ).tolist()
yield from np.random.randint(
high=n, size=(self.num_samples % 32,), dtype=np.int64
yield from flow.randint(
high=n,
size=(self.num_samples % 32,),
dtype=flow.int64,
generator=generator,
).tolist()
# yield from flow.randint(
# high=n,
# size=(self.num_samples % 32,),
# dtype=flow.int64,
# generator=generator,
# ).tolist()
else:
yield from flow.randperm(n, generator=generator).tolist()

Expand Down
18 changes: 4 additions & 14 deletions python/oneflow/utils/vision/transforms/transforms.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -586,11 +586,8 @@ def get_params(
if w == tw and h == th:
return 0, 0, h, w

# TODO:replace with flow.randint
# i = flow.randint(0, h - th + 1, size=(1, )).item()
# j = flow.randint(0, w - tw + 1, size=(1, )).item()
i = np.random.randint(low=0, high=h - th + 1, size=(1,), dtype=np.int32)
j = np.random.randint(low=0, high=w - tw + 1, size=(1,), dtype=np.int32)
i = flow.randint(0, h - th + 1, size=(1,)).item()
j = flow.randint(0, w - tw + 1, size=(1,)).item()
return i, j, th, tw

def __init__(
Expand Down Expand Up @@ -788,15 +785,8 @@ def get_params(
h = int(round(math.sqrt(target_area / aspect_ratio)))

if 0 < w <= width and 0 < h <= height:
# TODO:replace with flow.randint
# i = flow.randint(0, height - h + 1, size=(1,)).item()
# j = flow.randint(0, width - w + 1, size=(1,)).item()
i = np.random.randint(
low=0, high=height - h + 1, size=(1,), dtype=np.int32
)
j = np.random.randint(
low=0, high=width - w + 1, size=(1,), dtype=np.int32
)
i = flow.randint(0, height - h + 1, size=(1,)).item()
j = flow.randint(0, width - w + 1, size=(1,)).item()
return i, j, h, w

# Fallback to central crop
Expand Down

0 comments on commit d23f3a1

Please sign in to comment.