Skip to content

Commit

Permalink
Merge the documents from develop branch to master.
Browse files Browse the repository at this point in the history
  • Loading branch information
qiuwch committed Feb 23, 2017
1 parent 0d63501 commit 48d1d99
Show file tree
Hide file tree
Showing 26 changed files with 881 additions and 204 deletions.
13 changes: 13 additions & 0 deletions docs/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Convert ipynb notebook to markdown documentation
# An alternative way is linking to ipynb files directly
#
nbconvert=jupyter nbconvert --to markdown --template ./ipynb_template.tpl

ipynb:
${nbconvert} ../client/examples/generate-images.ipynb --output-dir ./ipynb

# Make sure no dead links in the doc.
check-docs:
cd .. && mkdocs build && linkchecker site

.PHONY: ipynb
1 change: 1 addition & 0 deletions docs/contact.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Team members
18 changes: 18 additions & 0 deletions docs/game/configuration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
!!! warning
This page is under construction

## The configuration file

Start from UnrealCV v0.3.1, the config of UnrealCV can be configured in a configuration file. Use `vget /unrealcv/status` to see current configuration.

## Change game resolution

The output resolution of UnrealCV is independent of the window size.

If you want to change the display resolution.
In game mode, use console command `r.setres 320x240`

When use `play -> selected viewport` the resolution can not be changed, use `play -> new window editor` instead.
<!-- TODO: Change resolution -->

## Change the server port
8 changes: 4 additions & 4 deletions docs/game/faster-rcnn.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,15 @@ This is a video showing connecting Faster-RCNN with a video game for diagnosis.
We are working on a python layer for Caffe and will include it in the next release.
</blockquote>

Make sure you read [getting started](/binary/getting-started.md) first and know how to use the compiled binary. Faster-RCNN is an algorithm to detect objects in an image. The original paper is published [here](http://arxiv.org/abs/1506.01497). In this tutorial we show how to run Faster-RCNN in the virtual scene. You can observe the Faster-RCNN can do a pretty good job, but still consistently making some mistakes, for example, detect photos as TV monitor.
Make sure you read [getting started](/getting-started.md) first and know how to use the compiled binary. Faster-RCNN is an algorithm to detect objects in an image. The original paper is published [here](http://arxiv.org/abs/1506.01497). In this tutorial we show how to run Faster-RCNN in the virtual scene. You can observe the Faster-RCNN can do a pretty good job, but still consistently making some mistakes, for example, detect photos as TV monitor.

## Run demo

To run the program illustrated in the video.

1. Install [py-faster-rcnn](https://github.com/rbgirshick/py-faster-rcnn). Follow the instruction to setup. Make sure you can run `tools/demo.py`

2. Run RealisticRendering in the [getting started tutorial](basics.md)
2. Run RealisticRendering in the [getting started tutorial](/getting-started.md)

3. Modify `rcnn_path` in the script `client/examples/faster-rcnn.py` to point to your Faster-RCNN installation, then run it.

Expand All @@ -29,6 +29,6 @@ python faster-rcnn.py

After running `faster-rcnn.py`, you are expected to see a message showing the client successfully connected to the game. When you clicked mouse in the game, a frame will be sent to Faster RCNN for detection. Using the same technique shown in this tutorial, algorithms used for depth estimation, object recognition can be also easily integrated.

Here we show how to do testing in a virtual scene. If you are interested in training model with synthetic images. Please refer to the tutorial about [generating images](game/generate-images.md).
Here we show how to do testing in a virtual scene. If you are interested in training model with synthetic images. Please refer to the tutorial about [generating images](/ipynb/generate-images.md).

If you want the training procedure can also interact with the scene, such as reinforcement learning, please [contact us](contact.md), we are working on it.
If you want the training procedure can also interact with the scene, such as reinforcement learning, please join the discussion [here](https://groups.google.com/d/topic/unrealcv/0zFjX_DCe7U/discussion), we are working on it.
1 change: 1 addition & 0 deletions docs/gen_ipynb.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
# Convert ipynb to documentation. Make sure no error in the ipython notebooks.
79 changes: 79 additions & 0 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
This page introduces *UnrealCV commands* and how to use them to perform basic tasks. We also show how to use a python script to control an UnrealCV embedded game through these commands.

## 1. Download a game binary

This tutorial will use a game binary to demonstrate UnrealCV commands. You can also [create your own game using UnrealCV plugin](plugin/usage.md).

First you need to download a game binary from [our model zoo](reference/model-zoo.md). For this tutorial, please download [RealisticRendering](reference/model-zoo.md#realistic_rendering). After unzip and run the binary, you are expected to see a screen like this. The game will be started in a window mode with resolution 640x480, you can change the resolution by [changing the configuration file](game/configuration.md) of UnrealCV.

<center>
<img alt="Startup Screenshot" src="/images/rr_init.png" width="300px"/>
<p>Fig.1. Initial screen of the game</p>
</center>

Use mouse to look around and use keys `w a s d` to navigate, use `q e` to level the camera up and down. If you want to release mouse cursor from the game, press <code>&#96;</code> (the key on top of tab).

## 2. UnrealCV commands

UnrealCV provides a set of commands for computer vision research. These commands are used to perform various tasks, such as control camera and get ground truth. The table below summaries commands used in this tutorial. The complete list can be found in [the command list](reference/command.md) in the reference section.

| Command | Help |
|:------------------------------------|:--------------------------------------------------|
| vset /viewmode [viewmode_name] | Set ViewMode to (lit, normal, depth, object_mask) |
| vget /camera/0/lit | Save image to disk and return filename |
| vset /camera/0/location [x] [y] [z] | Set camera location |

<h3>Try UnrealCV commands</h3>
Unreal Engine provides a built-in console to help developers to debug games. This built-in console is a convenient way of trying UnrealCV commands. To open the console, press
<code>&#96;</code>
(the key on top of tab) twice, a console will pop out, shown in Fig.2. Type in `vset /viewmode object_mask` you are expected to see the object instance mask. Use `vset /viewmode lit` to switch back to normal rendering setting.

<center>
<img alt="console" src="/images/console.png"/>
<p>Fig.2 Use console to try UnrealCV commands</p>
</center>

## 3. Use python client to execute commands

If we want to generate a large-scale synthetic dataset, or do active tasks, such as reinforcement learning, in this virtual world. We need to allow an intelligent agent to perceive, navigate and interact in the scene. We provide UnrealCV client to enable other programs to communicate with this virtual world. The client will use a [plain-text protocol](reference/arch.md#protocol) to exchange information with the game.

Here we use the python client for illustration. If you are looking for a MATLAB client, please see [here](reference/api.md#matlab).

First, we need to install the python client library.

<h3>Install UnrealCV python library</h3>
```shell
pip install unrealcv
```

<h3>Generate some images from the scene</h3>
```python
from unrealcv import client
client.connect() # Connect to the game
if not client.isconnected(): # Check if the connection is successfully established
print 'UnrealCV server is not running. Run the game from http://unrealcv.github.io first.'
else:
filename = client.request('vget /camera/0/lit')
filename = client.request('vget /camera/0/depth depth.exr')
```
You can find this example in [client/examples/10lines.py]().

If you encountered any errors following this tutorial, please see [the diagnosis]() page to find a solution.


## Next

<h3>Use UnrealCV in the game mode or plugin mode?</h3>

For the game mode, you can use a compiled game binary. You can freely control the camera in this game and generate images and ground truth from it. But it is not easy to change the scene, such as add more objects or change the material properties. If you have access to an UE4 project and know how to use the UE4Editor, you can install the plugin to UE4Editor, so that you can combine the power of UE4Editor and UnrealCV to create new virtual worlds for research.

<h3>Tutorials</h3>

- [How to generate an image dataset](ipynb/generate-images.md)
- [Integrate with a deep learning framework](game/faster-rcnn.md)
- [Use the plugin in UE4Editor](plugin/usage.md)
- [Modify code and add a new command](plugin/add-command.md)

<h3>Articles</h3>

- To fully understand how does UnrealCV work and the technical details, please read its [architecture](reference/arch.md) or [our paper](http://arxiv.org/abs/1609.01326). For a complete list of available commands, please read [the command list](reference/command.md) in the reference section.
Binary file added docs/images/plugin.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/realistic_rendering.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
60 changes: 10 additions & 50 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -1,55 +1,15 @@
## Introduction
UnrealCV helps computer vision researchers build virtual worlds using Unreal Engine 4 (UE4). It extends UE4 with a plugin by providing:

1. A set of UnrealCV commands to interact with the virtual world.
2. Communication between UE4 and an external program, such as Caffe.

![teaser](images/homepage_teaser.png)
Images generated from the technical demo [RealisticRendering](reference/model_zoo.html#realistic_rendering)

The goal of this documentation is to help you integrate UnrealCV with your research project.


## Documentation Outline

UnrealCV is an UE4 plugin, but it can be used in two modes and this documentation will show which mode is suitable for you and how to use it in these two modes.
<h1>Introduction</h1>

1. `Game` mode: use precompiled binaries provided by Unrealcv community. This is as simple as running a video game, no knowledge of Unreal Engine is required.
2. `Plugin` mode: use unrealcv plugin in UE4Editor. In this mode, UnrealCV is installed into UE4 as an editor plugin and can be used to build a virtual world.

Unrealcv consists two parts:
1. `unrealcv server` which is embedded into a video game (in game mode)
or in the UE4 editor (in plugin mode).
2. `unrealcv client` use commands to control the server.

This documentation is organized as follow:

- UnrealCV basics
- [Getting Started](basic/getting-started.md)
- Download and install
- [Game mode or plugin mode](basic/game_or_plugin.md)
- UnrealCV Commands

- Game Mode
- [Usage](game/usage.md)
- [Tutorial: Generate images](game/generate.md)
- [Tutorial: Integrate with other programs]()

- Plugin Mode
- [Usage](plugin/usage.md)
- [Install](plugin/usage.md#install)
- [Development](plugin/dev.md)
- [Environment setup](dev/setup.md)
- [Modify UnrealCV code to add more functionalities](dev/example-add-new-command.md)

- Reference
- Model Zoo
- [Architecture](reference/arch.md)
- [Command list](reference/command.md)
UnrealCV helps computer vision researchers build virtual worlds using Unreal Engine 4 (UE4). It extends UE4 with a plugin by providing:

1. a set of UnrealCV commands to interact with the virtual world
2. Communication between UE4 and an external program, such as Caffe or tensorflow.

## Contact
<center>
<img src='images/homepage_teaser.png'/>
<p>Fig.1. Images generated from the technical demo <a href=''>RealisticRendering</a></p>
</center>

This documentation is written with markdown, hosted in ReadTheDocs. All the markdown source files are hosted [here](https://github.com/unrealcv/unrealcv/tree/master/docs). To see the docs for different versions, please check this [versions](https://readthedocs.org/projects/unrealcv/versions/) page.
Currently, UnrealCV plugin supports UE4 (4.12, 4.13, 4.14) in Windows, Linux and Mac. UnrealCV client supports python and MATLAB (experimental). Read [getting started](getting-started.md) to learn how to use UnrealCV.

If you found any part of this documentation is unclear, please feel free to ask question in our gitter channel. You are welcome to contribute by making a pull request or report issues in [issue tracker](https://github.com/unrealcv/unrealcv/issues).
This doc is hosted in [readthedocs](https://unrealcv.readthedocs.io). For the doc of a specific version of unrealcv, please see [the version page](https://readthedocs.org/projects/unrealcv/versions/)

0 comments on commit 48d1d99

Please sign in to comment.