"import tools" in my codebase imports detectron2's tools rather than my own library.

[danielgordon10]

It's pretty common practice for pip libraries to put all their code inside a folder with the same name in order to namespace all packages to the outermost unique library name. In the detectron2 repo, there are a few folders that violate this, notably tools and projects. I would like to understand why they are set up this way rather than living inside the detectron2 folder. Is there a way I can fix this by modifying the setup.py process? Or will I have to move or rename the folders in my fork and risk merge issues down the line?

Instructions To Reproduce the Issue:

In my codebase

from tools import foo

Expected behavior:

Imports my script named foo living in <root>/tools/foo.py

Actual behavior

Python says "foo" is an unknown symbol

Environment:

----------------------  ----------------------------------------------------------------
sys.platform            linux
Python                  3.8.10 (default, Nov 26 2021, 20:14:08) [GCC 9.3.0]
numpy                   1.20.2
detectron2              0.6 @/home/daniel/detectron2/detectron2
detectron2._C           not built correctly: No module named 'detectron2._C'
Compiler ($CXX)         c++ (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
CUDA compiler           Build cuda_11.1.TC455_06.29190527_0
DETECTRON2_ENV_MODULE   <not set>
PyTorch                 1.9.0+cu111 @/usr/local/lib/python3.8/dist-packages/torch
PyTorch debug build     False
GPU available           Yes
GPU 0,1,2,3             NVIDIA GeForce RTX 2080 Ti (arch=7.5)
Driver version          470.86
CUDA_HOME               /usr/local/cuda
Pillow                  8.2.0
torchvision             0.10.0+cu111 @/usr/local/lib/python3.8/dist-packages/torchvision
torchvision arch flags  3.5, 5.0, 6.0, 7.0, 7.5, 8.0, 8.6
cv2                     3.4.13
----------------------  ----------------------------------------------------------------
PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.1.2 (Git Hash 98be7e8afa711dc9b66c8ff3504129cb82013cdb)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
  - CuDNN 8.0.5
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.9.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

Testing NCCL connectivity ... this should not hang.
NCCL succeeded.

[github-actions]

You've chosen to report an unexpected problem or bug. Unless you already know the root cause of it, please include details about it by filling the issue template.
The following information is missing: "Your Environment";

[ppwwyyxx]

Take https://github.com/numpy/numpy/tree/main/tools for example. It is not a wrong thing to have these directories.

I think the problem is either because

• you didn't install detectron2 properly through pip (but with hacks like PYTHONPATH).
  • in this case you should install it properly
• Or, you write new code inside detectron2's root directory
  • in this case you should develop your project in your own directory

In particular, it looks like pip install -e , i.e. the "in-place editable install" will lead to this issue. The regular install (without "-e") should work fine.

[danielgordon10]

We are installing via creating a wheel and then pip installing that wheel.
I actually noticed the change may have aligned with detectron2 adding the __init__.py inside the tools directory. The numpy example you posted does not have an init file in their tools directory. We did not see this problem before the __init__.py file was created.

[ppwwyyxx]

Thanks for the info. I don't know there is such a file and it looks like a hack to me from the log.
Having that file can explain the issue for tools, but I don't think projects will have similar issue.

For tools, I think adding it to the exclude here:

detectron2/setup.py

Line 154 in 710e779

packages=find_packages(exclude=("configs", "tests*")) + list(PROJECTS.keys()),

probably will fix it.

[xwang233]

Hi @ppwwyyxx , I'm pretty sure when I install numpy, they don't copy their tools folder to pollute the global python site-packages import path. See pytorch/benchmark#970

[ppwwyyxx]

@xwang233 I agree with you, and I think my comment above at #3934 (comment) will probably fix it

[gallegi]

Thanks for the info. I don't know there is such a file and it looks like a hack to me from the log. Having that file can explain the issue for tools, but I don't think projects will have similar issue.

For tools, I think adding it to the exclude here:

detectron2/setup.py

Line 154 in 710e779

packages=find_packages(exclude=("configs", "tests*")) + list(PROJECTS.keys()),

probably will fix it.

It does not work

[tkkcc]

you need to clean rebuild it, s.t. remove build and egg folder first.
and use pip install without -e
@gallegi