RT-1

This is the completion of google's rt-1 project code and can run directly.

You can view the google source code here: robotics_transformer

Using Method

1. 下载language-table数据集，详见下文Downloading the dataset，下载Universal Sentence Encoder模型

2. 通过language_table_data_reconstruction文件夹下代码对instrucion进行编码，原数据集为UTF-8编码格式

3. 运行前需要解决tensorflow版本兼容性问题，参见我的回答contrib_answer,运行distribute_train.py,保存模型

4. 使用language_table文件夹下代码进行测试，仿真环境详见language_table，整体流程如上图所示

Features

• Film efficient net based image tokenizer backbone
• Token learner based compression of input tokens
• Transformer for end to end robotic control
• Testing utilities

Getting Started

Downloading the dataset

RT-1 dataset: robotics_transformer_dataset

Language-table dataset: language_table_dataset

Both datasets are in RLDS format

Installation

Clone the repo

Before running, you need to resolve the tensorflow version compatibility issues in tensor2robot, see my answer contrib_answer

git clone https://github.com/RLC-Lab/Foundation-Models-for-Robots
# clone the repo rt-1 used
cd Foundation-Models-for-Robots
git clone https://github.com/google-research/tensor2robot

# Install protoc and compile the protobufs.
pip install protobuf
cd tensor2robot/proto
protoc -I=./ --python_out=`pwd` t2r.proto

# Optional: Create a conda env,you can also follow google's instructions for configuration
cd ../..
conda env create -f tf-rt1_environment.yaml

# Run distributed code
python -m robotics_transformer.distribute_train

Using trained checkpoints

Checkpoints are included in trained_checkpoints/ folder for three models:

1. RT-1 trained on 700 tasks
2. RT-1 jointly trained on EDR and Kuka data
3. RT-1 jointly trained on sim and real data

They are tensorflow SavedModel files. Instructions on usage can be found here

Future Releases

The current repository includes an initial set of libraries for early adoption. More components may come in future releases.

License

The Robotics Transformer library is licensed under the terms of the Apache license.

Acknowledgements

Special thanks to these people for their help in this project:kpertsch

Contact

The project will continue to improve and update, if you have any questions about the use of this project or suggestions for modification, please contact us by email 120l021822@stu.hit.edu.cn