- Adb tools
- Android Phone
site for installation)for IOS (Refer to this
Algorithms for Localization
- Multiscale search
- Fast search
- CNN-based coarse-to-fine model
For algorithm details, please go to https://zhuanlan.zhihu.com/p/32636329.
Notice: CV based fast-search only support Android for now
Before running our code, connect to your phone via USB.
If Android phone, open the USB debugging at developer options enter
adb devices to ensure that the list is not empty.
If iPhone, please ensure that you have a mac. Then following this link for preparation.
It is recommended to download the pre-trained model following the link below and run the following code
python nn_play.py --phone Android --sensitivity 2.045
You can also try
play.py by running the following code
python play.py --phone Android --sensitivity 2.045
--phonehas two options: Android or IOS.
--sensitivityis the constant parameter that controls the pressing time.
nn_play.pyuses CNN-based coarse-to-fine model, supporting Android and IOS (more robust)
play.pyuses multiscale search and fast search algorithms, supporting Android and IOS (it may fail sometimes in other phones)
Our method can correctly detect the positions of the man (green dot) and the destination (red dot).
It is easy to reach the state of art as long as you like. But I choose to go die after 859 jumps for about 1.5 hours.
Here is a video demo. Excited!
Train Log & Data
CNN train log and train&validation data avaliable at
Training: download and untar data into any directory, and then modify
self.data_dir in those files under
Inference: download and unzip train log dirs(
How to Train CNN models by yourself?
- Download and untar data into any directory, and then modify
self.data_dirin those files under
base.largeis model dir for coarse model,
base.fineis model dir for fine model, other dirs under
cnn_coarse_to_fine/configare models we don't use, but if you have interests, you can try train other models by yourself.
python3 train.py -g 0to train your model,
-gto specify GPU to use, if you don't have GPU, training model is not recommended because training speed with CPU is very slow.
- After training, move or copy
.ckptfile to train log dirs(
train_logs_fine) for use.