-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature/enet image segmenter #784
Merged
Merged
Changes from 3 commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
77b560a
Initial Release of ENEt Image Segmenter
amc-nu 24bccba
Node Publishes
amc-nu bba97be
Removed unused params
amc-nu 530f787
Added More Instructions to README
amc-nu 4cb094d
Added extra instructions
amc-nu b701218
Package renamed to image_segmenter
amc-nu 171e604
Merge branch 'develop' into feature/enet_image_segmenter
amc-nu b27a75f
Modified package README to reflect updated naming
amc-nu 852ab6a
Fixed dependencies
amc-nu 75547d6
Added curly braces to one line ifs
amc-nu 082eb61
Added Further compilation instructions
amc-nu 9c2bcbc
Path in readme file updated
amc-nu File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
92 changes: 92 additions & 0 deletions
92
ros/src/computing/perception/detection/packages/enet_image_segmenter/CMakeLists.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,92 @@ | ||
cmake_minimum_required(VERSION 2.8.12) | ||
project(enet_image_segmenter) | ||
execute_process( | ||
COMMAND rosversion -d | ||
OUTPUT_VARIABLE ROS_VERSION | ||
OUTPUT_STRIP_TRAILING_WHITESPACE | ||
) | ||
|
||
include(FindPkgConfig) | ||
|
||
if ("${ROS_VERSION}" MATCHES "(indigo|jade)") | ||
FIND_PACKAGE(catkin REQUIRED COMPONENTS | ||
cv_bridge | ||
image_transport | ||
roscpp | ||
sensor_msgs | ||
std_msgs | ||
autoware_msgs | ||
) | ||
elseif ("${ROS_VERSION}" MATCHES "(kinetic)") | ||
FIND_PACKAGE(catkin REQUIRED COMPONENTS | ||
cv_bridge | ||
image_transport | ||
roscpp | ||
std_msgs | ||
autoware_msgs | ||
sensor_msgs | ||
) | ||
endif() | ||
|
||
FIND_PACKAGE(CUDA) | ||
FIND_PACKAGE(OpenCV REQUIRED) | ||
|
||
EXECUTE_PROCESS( | ||
COMMAND uname -m | ||
OUTPUT_VARIABLE ARCHITECTURE | ||
OUTPUT_STRIP_TRAILING_WHITESPACE | ||
) | ||
|
||
if ("${ROS_VERSION}" MATCHES "(indigo|jade)") | ||
catkin_package( | ||
CATKIN_DEPENDS std_msgs geometry_msgs autoware_msgs | ||
) | ||
elseif ("${ROS_VERSION}" MATCHES "(kinetic)") | ||
catkin_package( | ||
CATKIN_DEPENDS std_msgs geometry_msgs autoware_msgs | ||
) | ||
endif() | ||
########### | ||
## Build ## | ||
########### | ||
|
||
SET(CMAKE_CXX_FLAGS "-std=c++11 -O2 -g -Wall ${CMAKE_CXX_FLAGS}") | ||
|
||
INCLUDE_DIRECTORIES( | ||
${catkin_INCLUDE_DIRS} | ||
) | ||
|
||
|
||
#####ENET######## | ||
##############################SSD'sFORK of CAFFE NEEDS TO BE PREVIOUSLY COMPILED#################### | ||
set(ENET_CAFFE_PATH "$ENV{HOME}/ENet/caffe-enet/distribute") | ||
#################################################################################################### | ||
if(EXISTS "${ENET_CAFFE_PATH}") | ||
|
||
ADD_EXECUTABLE(enet_image_segmenter | ||
src/enet_image_segmenter_node.cpp | ||
src/enet_image_segmenter.cpp | ||
) | ||
|
||
TARGET_LINK_LIBRARIES(enet_image_segmenter | ||
${catkin_LIBRARIES} | ||
${OpenCV_LIBS} | ||
${CUDA_LIBRARIES} | ||
${CUDA_CUBLAS_LIBRARIES} | ||
${CUDA_curand_LIBRARY} | ||
${ENET_CAFFE_PATH}/lib/libcaffe.so | ||
glog | ||
) | ||
|
||
TARGET_INCLUDE_DIRECTORIES(enet_image_segmenter PRIVATE | ||
${CUDA_INCLUDE_DIRS} | ||
${ENET_CAFFE_PATH}/include | ||
include | ||
) | ||
|
||
ADD_DEPENDENCIES(enet_image_segmenter | ||
${catkin_EXPORTED_TARGETS} | ||
) | ||
else() | ||
message("'ENet/Caffe' is not installed. 'enet_image_segmenter' will not be built.") | ||
endif() |
28 changes: 28 additions & 0 deletions
28
ros/src/computing/perception/detection/packages/enet_image_segmenter/README.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
# How to install Caffe for ENet | ||
|
||
1. Complete the [pre-requisites](http://caffe.berkeleyvision.org/install_apt.html) | ||
|
||
2. Clone ENEt Caffe fork in your home directory | ||
``` | ||
% git clone https://github.com/TimoSaemann/ENet.git | ||
``` | ||
|
||
3. Follow the authors' instruction to complete the requisites to compile. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please add authors' instruction link There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
|
||
4. Compile ENet fork of Caffe | ||
``` | ||
make && make distribute | ||
``` | ||
|
||
6. Download pretrained models as pointed in https://github.com/TimoSaemann/ENet/tree/master/Tutorial#kick-start or use your own. | ||
|
||
If you didn't install ENet Caffe in `ENet` inside your home, modify the ENet's CMakeFiles and point them to your directories. | ||
|
||
Once compiled, run from the terminal, or launch from RunTimeManager | ||
|
||
``` | ||
% roslaunch enet_image_segmenter enet_image_segmenter.launch | ||
``` | ||
Remember to modify the launch file located inside | ||
`computing/perception/detection/packages/enet_image_segmenter/launch/enet_image_segmenter.launch` | ||
and point the network, pretrained models and LUT file to your paths. |
52 changes: 52 additions & 0 deletions
52
...uting/perception/detection/packages/enet_image_segmenter/include/enet_image_segmenter.hpp
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,52 @@ | ||
/* | ||
* enet_image_segmenter.hpp | ||
* | ||
* Created on: Aug 23, 2017 | ||
* Author: ne0 | ||
*/ | ||
|
||
#ifndef ENET_IMAGE_SEGMENTER_HPP_ | ||
#define ENET_IMAGE_SEGMENTER_HPP_ | ||
|
||
#include <caffe/caffe.hpp> | ||
|
||
#include <opencv2/core/core.hpp> | ||
#include <opencv2/highgui/highgui.hpp> | ||
#include <opencv2/imgproc/imgproc.hpp> | ||
|
||
#include <algorithm> | ||
#include <iosfwd> | ||
#include <memory> | ||
#include <string> | ||
#include <utility> | ||
#include <vector> | ||
|
||
using namespace caffe; | ||
using std::string; | ||
|
||
class ENetSegmenter | ||
{ | ||
public: | ||
ENetSegmenter(const string& in_model_file, const string& in_trained_file, const string& in_lookuptable_file); | ||
|
||
void Predict(const cv::Mat& in_img, cv::Mat& out_segmented); | ||
|
||
private: | ||
void SetMean(const string& in_mean_file); | ||
|
||
void WrapInputLayer(std::vector<cv::Mat>* in_input_channels); | ||
|
||
void Preprocess(const cv::Mat& img, std::vector<cv::Mat>* in_input_channels); | ||
|
||
cv::Mat Visualization(cv::Mat in_prediction_map, string in_lookuptable_file); | ||
|
||
private: | ||
shared_ptr<Net<float> > net_; | ||
cv::Size input_geometry_; | ||
int num_channels_; | ||
string lookuptable_file_; | ||
cv::Scalar pixel_mean_; | ||
|
||
}; | ||
|
||
#endif /* ENET_IMAGE_SEGMENTER_HPP_ */ |
18 changes: 18 additions & 0 deletions
18
...ing/perception/detection/packages/enet_image_segmenter/launch/enet_image_segmenter.launch
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
<launch> | ||
<!-- arguments list --> | ||
<arg name="network_definition_file" default="$(env HOME)/ENet/prototxts/enet_deploy_final.prototxt"/> | ||
<arg name="pretrained_model_file" default="$(env HOME)/ENet/enet_weights_zoo/cityscapes_weights.caffemodel"/> | ||
<arg name="lookuptable_file" default="$(env HOME)/ENet/scripts/cityscapes19.png"/> | ||
|
||
<arg name="camera_id" default="/" /> | ||
<arg name="image_src" default="/image_raw"/> | ||
|
||
<!-- ENET --> | ||
<node pkg="enet_image_segmenter" name="enet_image_segmenter" type="enet_image_segmenter" output="screen"> | ||
<param name="network_definition_file" type="str" value="$(arg network_definition_file)"/> | ||
<param name="pretrained_model_file" type="str" value="$(arg pretrained_model_file)"/> | ||
<param name="lookuptable_file" type="str" value="$(arg lookuptable_file)"/> | ||
<param name="image_raw_node" type="str" value="$(arg camera_id)$(arg image_src)"/> | ||
</node> | ||
|
||
</launch> |
17 changes: 17 additions & 0 deletions
17
ros/src/computing/perception/detection/packages/enet_image_segmenter/package.xml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
<?xml version="1.0"?> | ||
<package> | ||
<name>enet_image_segmenter</name> | ||
<version>1.0.0</version> | ||
<description>Image Segmentation using ENet</description> | ||
<maintainer email="abrahammonrroy@yahoo.com">Abraham Monrroy</maintainer> | ||
<license>BSD</license> | ||
<buildtool_depend>catkin</buildtool_depend> | ||
<build_depend>std_msgs</build_depend> | ||
<build_depend>geometry_msgs</build_depend> | ||
<build_depend>autoware_msgs</build_depend> | ||
<run_depend>std_msgs</run_depend> | ||
<run_depend>geometry_msgs</run_depend> | ||
<run_depend>autoware_msgs</run_depend> | ||
<export> | ||
</export> | ||
</package> |
141 changes: 141 additions & 0 deletions
141
...computing/perception/detection/packages/enet_image_segmenter/src/enet_image_segmenter.cpp
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,141 @@ | ||
/* | ||
* enet_image_segmenter_node.cpp | ||
* | ||
* Created on: Aug 23, 2017 | ||
* Author: amc | ||
*/ | ||
|
||
#include "enet_image_segmenter.hpp" | ||
|
||
ENetSegmenter::ENetSegmenter(const string& in_model_file, | ||
const string& in_trained_file, | ||
const string& in_lookuptable_file) | ||
{ | ||
|
||
Caffe::set_mode(Caffe::GPU); | ||
|
||
/* Load the network. */ | ||
net_.reset(new Net<float>(in_model_file, TEST)); | ||
net_->CopyTrainedLayersFrom(in_trained_file); | ||
|
||
CHECK_EQ(net_->num_inputs(), 1)<< "Network should have exactly one input."; | ||
CHECK_EQ(net_->num_outputs(), 1)<< "Network should have exactly one output."; | ||
|
||
Blob<float>* input_layer = net_->input_blobs()[0]; | ||
num_channels_ = input_layer->channels(); | ||
CHECK(num_channels_ == 3 || num_channels_ == 1) << "Input layer should have 1 or 3 channels."; | ||
input_geometry_ = cv::Size(input_layer->width(), input_layer->height()); | ||
|
||
lookuptable_file_ = in_lookuptable_file; | ||
|
||
pixel_mean_ = cv::Scalar(102.9801, 115.9465, 122.7717); | ||
} | ||
|
||
void ENetSegmenter::Predict(const cv::Mat& in_image_mat, cv::Mat& out_segmented) | ||
{ | ||
Blob<float>* input_layer = net_->input_blobs()[0]; | ||
input_layer->Reshape(1, num_channels_, input_geometry_.height, | ||
input_geometry_.width); | ||
/* Forward dimension change to all layers. */ | ||
net_->Reshape(); | ||
|
||
std::vector<cv::Mat> input_channels; | ||
WrapInputLayer(&input_channels); | ||
|
||
Preprocess(in_image_mat, &input_channels); | ||
|
||
net_->Forward(); | ||
|
||
/* Copy the output layer to a std::vector */ | ||
Blob<float>* output_layer = net_->output_blobs()[0]; | ||
|
||
int width = output_layer->width(); | ||
int height = output_layer->height(); | ||
int channels = output_layer->channels(); | ||
int num = output_layer->num(); | ||
|
||
// compute argmax | ||
cv::Mat class_each_row(channels, width * height, CV_32FC1, | ||
const_cast<float *>(output_layer->cpu_data())); | ||
class_each_row = class_each_row.t(); // transpose to make each row with all probabilities | ||
cv::Point maxId; // point [x,y] values for index of max | ||
double maxValue; // the holy max value itself | ||
cv::Mat prediction_map(height, width, CV_8UC1); | ||
for (int i = 0; i < class_each_row.rows; i++) | ||
{ | ||
minMaxLoc(class_each_row.row(i), 0, &maxValue, 0, &maxId); | ||
prediction_map.at<uchar>(i) = maxId.x; | ||
} | ||
|
||
out_segmented = Visualization(prediction_map, lookuptable_file_); | ||
|
||
cv::resize(out_segmented, out_segmented, cv::Size(in_image_mat.cols, in_image_mat.rows)); | ||
|
||
|
||
} | ||
|
||
cv::Mat ENetSegmenter::Visualization(cv::Mat in_prediction_map, string in_lookuptable_file) | ||
{ | ||
|
||
cv::cvtColor(in_prediction_map.clone(), in_prediction_map, CV_GRAY2BGR); | ||
cv::Mat label_colours = cv::imread(in_lookuptable_file, 1); | ||
cv::cvtColor(label_colours, label_colours, CV_RGB2BGR); | ||
cv::Mat output_image; | ||
LUT(in_prediction_map, label_colours, output_image); | ||
|
||
return output_image; | ||
} | ||
|
||
void ENetSegmenter::WrapInputLayer(std::vector<cv::Mat>* in_input_channels) | ||
{ | ||
Blob<float>* input_layer = net_->input_blobs()[0]; | ||
|
||
int width = input_layer->width(); | ||
int height = input_layer->height(); | ||
float* input_data = input_layer->mutable_cpu_data(); | ||
for (int i = 0; i < input_layer->channels(); ++i) | ||
{ | ||
cv::Mat channel(height, width, CV_32FC1, input_data); | ||
in_input_channels->push_back(channel); | ||
input_data += width * height; | ||
} | ||
} | ||
|
||
void ENetSegmenter::Preprocess(const cv::Mat& in_image_mat, | ||
std::vector<cv::Mat>* in_input_channels) | ||
{ | ||
/* Convert the input image to the input image format of the network. */ | ||
cv::Mat sample; | ||
if (in_image_mat.channels() == 3 && num_channels_ == 1) | ||
cv::cvtColor(in_image_mat, sample, cv::COLOR_BGR2GRAY); | ||
else if (in_image_mat.channels() == 4 && num_channels_ == 1) | ||
cv::cvtColor(in_image_mat, sample, cv::COLOR_BGRA2GRAY); | ||
else if (in_image_mat.channels() == 4 && num_channels_ == 3) | ||
cv::cvtColor(in_image_mat, sample, cv::COLOR_BGRA2BGR); | ||
else if (in_image_mat.channels() == 1 && num_channels_ == 3) | ||
cv::cvtColor(in_image_mat, sample, cv::COLOR_GRAY2BGR); | ||
else | ||
sample = in_image_mat; | ||
|
||
cv::Mat sample_resized; | ||
if (sample.size() != input_geometry_) | ||
cv::resize(sample, sample_resized, input_geometry_); | ||
else | ||
sample_resized = sample; | ||
|
||
cv::Mat sample_float; | ||
if (num_channels_ == 3) | ||
{ | ||
sample_resized.convertTo(sample_float, CV_32FC3); | ||
} | ||
else | ||
{ | ||
sample_resized.convertTo(sample_float, CV_32FC1); | ||
} | ||
|
||
cv::split(sample_float, *in_input_channels); | ||
|
||
CHECK(reinterpret_cast<float*>(in_input_channels->at(0).data) | ||
== net_->input_blobs()[0]->cpu_data()) | ||
<< "Input channels are not wrapping the input layer of the network."; | ||
} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since caffe-enet is contained in submodule, recursive option is necessary.
git clone --recursive https://github.com/TimoSaemann/ENet.git
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@yk-fujii Done