readme updated, new fusion segmentation added

README.md

@@ -17,19 +17,25 @@ Research and develop a deep learning-driven self-driving car. The vehicle should
 2. Semantic Segmentation
 3. Object Detection 🚙
 4. Drive by Wire (DBW)
-5. Traffic Light Detection 🚦
-6. Lane Detection 🛣
-7. Localization 🛰️ (currently with GPS)
+5. Localization 🛰️ (currently with GPS)
+6. CARLA simulator integration 
 
-Path planning is coming soon...
+Stereoscopic vision system coming soon...
 
 For the full documentation of the development process, please visit my website: [www.neilnie.com](https://www.neilnie.com)
 
 # Try it out
+Before you jump in, let me describe the hardware requirement for this project. **A webcam is the minimum requirment.** At this point, you can only run the whole system on the actual self-driving vehicle. ROS will throw warnings (even errors) at you if you don't have the hardware connected to your Linux machine. **If you don't have access to the hardware setup, don't worry** 👇
+
+- You can tryout individual packages and nodes, and might find them helpful for your own projects. 
+- You can also tryout the CARLA simulator. (Maybe even improve the current system.)
+
+To compile the project:
+
 1. Clone the repository.
 2. Make sure you have all the [dependencies](./ros/requirements.txt) installed. 
-3. Make sure that you have [ROS](http://wiki.ros.org/melodic/Installation/Ubuntu) installed on your computer. (I am currently using ROS Melodic)
-4. `cd PROJECT_DIRECTORY/ros`
+3. Make sure that you have [ROS](http://wiki.ros.org/melodic/Installation/Ubuntu) installed on your computer. (I am using ROS Melodic)
+4. `cd PROJECT_DIRECTORY/ros` 
 5. `catkin_make`
 6. `source devel/setup.bash`
 7. `roslaunch driver drive.launch`
@@ -45,7 +51,6 @@ You should see this screen pop up.
 # ROS
 This project uses ROS. The launch files will launch the neccesary nodes as well as rviz for visualization. __For more information on ROS, nodes, topics and others please refer to the ROS [README](./ros/README.md).__
 
-
 # Simulation
 (🏗 Construction Zone 🚧)
 

ros/src/segmentation/scripts/README.md

@@ -3,56 +3,3 @@
 🚧 This is still under development 🏗
 
 Semantic segmentation has become an important component of self-driving vehicles. It allows the car to understand the surroundings by classifying every pixel of the input image.
-
-![](./media/image-1.png)
-
-## Running...
-To run inference on the pre-trained models, please use `segmentor.py`. 
-
-	from segmentor import Segmentor
-	seg = Segmentor()
-	cls_output, img_viz = seg.semantic_segmentation(image=image, visualization=True)
-
-`cls_output` is the pixel-wise classification result for all the categories. `img_viz` is a RGB image generated based on `cls_output`
-
-The best way to run some actual test is using `test.py`. You must specify the image path by changing the `path` variable. 
-
-The pre-trained weights are stored in the `./weights` directory.
-
-## Training
-
-`train.py` is the training script. `utils.py` contains all the categories (classes). You can modify them based on your dataset. 
-
-*training doesn't work very well.* I trying to fix the issue. 
-
-## Models
-
-The goal is to achieve real time semantic segmentation. Therefore, choosing & developing the appropriate model is critical. 
-I experimented with UNet and ENet, but realized drawbacks to both of these architectures. I eventually landed on ICNet, which
-stands for image cascade network
-
-![](./media/model_overview.png)
-
-Here is a simple benchmark comparison between ICNet and other popular semantic segmentation models. These images and visualizations
-are from the original ICNet paper, which can be found [here](https://arxiv.org/abs/1704.08545).  
-
-<center>
-<img src="./media/model_comparison.png" alt="image" width="500">
-</center>
-
-## About
-
-This project is created for the self-driving golf cart project that I have been working on. For more information on that, please refer to the [Github page](https://github.com/xmeng17/self-driving-golf-cart), or my [website](https://neilnie.com/the-self-driving-golf-cart-project/).
-
-If you have questions, comments or concerns, please contact me at [contact@neilnie.com](mailto:contact@neilnie.com). 
-
-## Others
-
-#### Details about trained models
-`./weights/new-enet-5.h5`: 512x512x26
-
-`./weights/enet-c-v1-2.h5`: 640x320x16
-
-`./weights/enet-c-v2-1.h5`: 1024x512x16
-
-The channel value correspond to the number of classes.

ros/src/segmentation/scripts/test_fusion.py

@@ -0,0 +1,105 @@
+#!/usr/bin/python
+
+import argparse
+import time
+import numpy as np
+import matplotlib.pyplot as plt
+from monodepth.monodepth_runner import monodepth_runner
+import utils
+import cv2
+
+from models.icnet_fusion import ICNet
+import configs
+
+#### Test ####
+
+# define global variables
+model_type = 'cross_fusion'
+checkpoint_path = '/home/neil/Workspace/semantic-segmentation/monodepth/models/cityscape/model_cityscapes.data-00000-of-00001'
+model_path = 'icnet_' + model_type + '_030_0.869.h5'
+test_img_path = "./testing_imgs/731.png"
+
+# ==== create monodepth runner ====
+depth_runner = monodepth_runner(checkpoint_path)
+
+# ====== Model ======
+net = ICNet(width=configs.img_width, height=configs.img_height, n_classes=34, weight_path="output/" + model_path,
+            mode=model_type)
+print(net.model.summary())
+
+
+def visualization_result(y, mid):
+    y = cv2.resize(y, (configs.img_width / 2, configs.img_height / 2))
+    image = utils.convert_class_to_rgb(y, threshold=0.50)
+    viz = cv2.addWeighted(mid, 0.8, image, 0.8, 0)
+    plt.figure(1)
+    plt.imshow(viz)
+    plt.show()
+
+    cv2.imwrite('seg_result_overlay.png', cv2.resize(cv2.cvtColor(viz, cv2.COLOR_RGB2BGR), (1024, 512)))
+
+
+def test_fusion():
+
+    # ======== Testing ========
+    x = cv2.resize(cv2.imread(test_img_path, 1), (configs.img_width, configs.img_height))
+    x = cv2.cvtColor(x, cv2.COLOR_BGR2RGB)
+    x_depth = depth_runner.run_depth(image_path=test_img_path, out_height=configs.img_height,
+                                     out_width=configs.img_width)
+    x_depth = np.dstack((x_depth, x_depth, x_depth))
+
+    mid = cv2.resize(x, (configs.img_width / 2, configs.img_height / 2))
+
+    X_color = np.zeros((1, configs.img_height, configs.img_width, 3), dtype='float32')
+    X_depth = np.zeros((1, configs.img_height, configs.img_width, 3), dtype='float32')
+
+    X_color[0, :, :, :] = x
+    X_depth[0, :, :, :] = x_depth
+
+    y = net.model.predict([X_color, X_depth], verbose=1)[0]
+
+    # ====== running... ======
+    start_time = time.time()
+    for i in range(10):
+        y = net.model.predict([X_color, X_depth])[0]
+
+    duration = time.time() - start_time
+    print('Generated segmentations in %s seconds -- %s FPS' % (duration / 10, 1.0 / (duration / 10)))
+
+    visualization_result(y=y, mid=mid)
+
+
+def test_early_fusion():
+
+    # ======== Testing ========
+    x = cv2.resize(cv2.imread(test_img_path, 1), (configs.img_width, configs.img_height))
+    x = cv2.cvtColor(x, cv2.COLOR_BGR2RGB)
+    x_depth = depth_runner.run_depth(image_path=test_img_path, out_height=configs.img_height,
+                                     out_width=configs.img_width)
+    x_depth = np.dstack((x_depth, x_depth, x_depth))
+
+    plt.imshow(x_depth)
+    plt.show()
+
+    mid = cv2.resize(x, (configs.img_width / 2, configs.img_height / 2))
+    x = np.array([np.concatenate((x, x_depth), axis=2)])
+
+    y = net.model.predict(x)[0]
+
+    # ===== running... =====
+    start_time = time.time()
+    for i in range():
+        y = net.model.predict(x)[0]
+
+    duration = time.time() - start_time
+    print('Generated segmentations in %s seconds -- %s FPS' % (duration / 10, 1.0 / (duration / 10)))
+
+    visualization_result(y=y, mid=mid)
+
+
+if __name__ == "__main__":
+
+    if model_type == 'mid_fusion' or model_type == 'cross_fusion':
+        test_fusion()
+    elif model_type == 'early_fusion':
+        test_early_fusion()