-
Notifications
You must be signed in to change notification settings - Fork 117
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The problem about image resize and image distortion correction #22
Comments
Hi, I am not sure whether I understand your questions correctly.
Let me know if this does not answer your questions. |
Yes, as long as the central generic model is used, the algorithm that I tried to describe with pseudo-code should achieve exactly this. If the non-central generic model is used, then this cannot be done like this in general however. |
hi@puzzlepaint , I'm still not understand, |
Notice that the successive Un-projection and Projection operations in the undistortion algorithm given above operate with different camera models. If you use the same model for both, of course the same pixel as the input should be returned (otherwise, there would be a problem with the model). However, in the undistortion algorithm, the un-projection would use the pinhole model which was (arbitrarily) chosen as undistortion target, and the projection would use the calibrated generic model. There is no implementation of a pinhole camera model in the provided demo code, but it is very simple to implement based on the focal length and projection center parameters. |
Hi @puzzlepaint, I am wondering how the pseudo-code achieve the undistortion.
I am not sure what "this pinhole calibration" refers to, is it the undistorted image? My understanding is: pixel p is the undistorted position with unknown color, after un-projection (pinhole) and projection (central-generic) we find its corresponding pixel p2 in the original image to fill the color. If it's true, then the pinhole model does the undistortion (in step a) instead of the central-generic model. |
Yes, it is the pinhole model that was chosen for the undistorted image.
I don't think so. Pixel p is supposed to be in the undistorted (pinhole) image, so the first step needs to be to apply the inverse pinhole camera matrix (i.e., unproject it). This returns the direction corresponding to this pixel. Then the direction can be projected to the distorted image using the calibrated generic model in order to look up the color.
That sounds correct to me.
That also sounds correct to me if replacing "undistortion" by "unprojection". Step a in the algorithm does refer to the pinhole model (maybe that was not sufficiently clear in the original formulation). |
Thanks for your reply @puzzlepaint. Now I'll say that there is only one divergence left: can the pixel p in the undistorted image be directly un-projected? Here is how I understand the imaging in the pinhole model (as described in OpenCV),
I believe that we can get the direction after un-projecting p_d instead of p by applying the inverse pinhole camera matrix. Could you give more details about how to "unproject it"? |
The undistorted image is supposed to not have any distortion. This is why the "pinhole model" chosen for the undistorted image should have no distortion coefficients. It will only have focal length parameters (f_x, f_y) and optical center parameters (c_x, c_y). Thus, no distortion handling is necessary on the side of the undistorted image. The unprojection is simply done by applying the inverse pinhole matrix to pixel (p.x, p.y)^T to get the direction ((p.x - c_x) / f_x, (p.y - c_y) / f_y, 1)^T. In the pseudo-code algorithm, the undistorted model and the calibrated generic model are two completely independently chosen / calibrated models. This differs from the two steps you cited, where, if I understand it correctly, seemingly the distortion parameters of a calibrated parametric model may be inversely applied to get some undistorted image. This cannot be used for a generic model (since it has no separate pinhole and distortion parameters), but the algorithm presented in pseudo-code above works just fine. |
Could you explain how the direction makes sense? If I understand your paper correctly, the calibrated directions are related to original pixels instead of undistorted pixels. I think that pixel p should be distorted to p_d based on distortion coefficients. Then applying: p_w = H^(-1) * p_d, the direction starts at (0, 0, 0) and points to p_w. |
This direction makes sense because it only depends on the arbitrarily chosen (!) pinhole intrinsics of the desired undistorted image. It has nothing to do with the generic calibration. In general, starting from a calibrated central camera model, you can choose any kind of arbitrary other central camera model, and an arbitrary orientation for it, and reproject the pixels from the calibrated model into the arbitrary other model if both are located at the same optical center. (This is because central camera models basically just represent observation directions that start from a single point, the optical center. Reprojecting an image into a different central model just re-samples the image pixels at arbitrary different directions.) With typical parametric camera models, they happen to be defined in a way that the distortion component and the "pixel mapping" component (with the f_x, f_y, c_x, c_y parameters) can be applied separately. But there is no need to restrict oneself to these same f_x, f_y, c_x, c_y parameters, and the same orientation, for the undistortion. I would suggest to simply try out the proposed undistortion algorithm, then you should see that it works correctly. |
Hi @puzzlepaint, I've tried the pseudo-code on the Tango data you provided at #46. Here is what I've done:
void bilinear_interpolate(const cv::Mat &src_img, const double src_x, const double src_y, uchar &color) {
double src_x_f = floor(src_x);
double src_x_c = ceil(src_x);
double src_y_f = floor(src_y);
double src_y_c = ceil(src_y);
double w1 = (src_x_c - src_x) / (src_x_c - src_x_f);
double w2 = (src_x - src_x_f) / (src_x_c - src_x_f);
double w3 = (src_y_c - src_y) / (src_y_c - src_y_f);
double w4 = (src_y - src_y_f) / (src_y_c - src_y_f);
uchar q12 = src_img.at<uchar>(src_y_f, src_x_f);
uchar q22 = src_img.at<uchar>(src_y_f, src_x_c);
uchar q11 = src_img.at<uchar>(src_y_c, src_x_f);
uchar q21 = src_img.at<uchar>(src_y_c, src_x_c);
color = w3 * (w1 * q12 + w2 * q22) + w4 * (w1 * q11 + w2 * q21);
}
bool test_undistortion() {
// Load the central_generic model.
const char* file_path = "/Users/rankencheng/Downloads/raxel/tango/result_15px_central_generic_20/intrinsics0.yaml";
CentralGenericCamera<double> camera;
std::string error_reason;
bool result = camera.Read(file_path, &error_reason);
if (!result) {
std::cout << "Reading the camera failed! Reason: " << error_reason << std::endl;
return false;
}
// Load fx, fy, cx, cy from the central_opencv model.
double parameters[12] = { 277.43476524575, 277.29416182849, 307.81265362247, 233.0392096016, 0.93640031669187, 0.35692232820102, 0.015722514959806, 1.2608235807082, 0.59007496453161, 0.079399687311869, -0.00027415199228906, 5.6684176272052e-05 };
const double& fx = parameters[0];
const double& fy = parameters[1];
const double& cx = parameters[2];
const double& cy = parameters[3];
cv::Mat src_img = cv::imread("/Users/rankencheng/Downloads/raxel/tango/image000000_f.png", cv::IMREAD_GRAYSCALE);
cv::Mat dst_img(src_img.rows * 2, src_img.cols * 2, CV_8U); // double the width and height to obtain larger FOV
const int x_begin = 0 - camera.width() * 0.5;
const int x_end = camera.width() * 1.5;
const int y_begin = 0 - camera.height() * 0.5;
const int y_end = camera.height() * 1.5;
// For each pixel p in this pinhole calibration.
for (int x = x_begin; x < x_end; ++x) {
for (int y = y_begin; y < y_end; ++y) {
// Un-project the pixel coordinate of p to a direction d.
// The unprojection is simply done by applying the inverse pinhole matrix to pixel (p.x, p.y)^T to get the direction ((p.x - c_x) / f_x, (p.y - c_y) / f_y, 1)^T.
CentralGenericCamera<double>::PointT d((x - cx) / fx, (y - cy) / fy, 1);
// Project this direction d to a pixel coordinate p2 using the calibrated camera model.
CentralGenericCamera<double>::PixelT p2;
camera.Project(d, &p2);
// Interpolate the pixel color at the obtained pixel coordinate p2 in the original image, and set the current pixel p in the undistorted image to this color.
// Bilinear interpolation is used.
uchar color;
double src_x = p2(0, 0);
double src_y = p2(1, 0);
if (src_x < 0 || src_x >= src_img.cols) continue;
if (src_y < 0 || src_y >= src_img.rows) continue;
bilinear_interpolate(src_img, src_x, src_y, color);
dst_img.at<uchar>(y + camera.height() * 0.5, x + camera.width() * 0.5) = color;
}
}
cv::imwrite("/Users/rankencheng/Downloads/raxel/test.png", dst_img);
return true;
} It seems that the algorithm works! By the way, I was wrong about the calibrated directions so that I made the mistake:
The direction ((p.x - c_x) / f_x, (p.y - c_y) / f_y, 1)^T makes sense because it's exact the resulted point (this point can be regarded as the direction) by applying the inverse camera intrinsics matrix to the pixel. I will conclude that the pixel to be un-projected should be the undistorted one instead of the distorted one in the pinhole model, while the pixel after projection should become distorted in the generic model. Thank you very much! |
Hi, @painterdrown |
The |
Hi @puzzlepaint, @painterdrown, I have cached the result of the unprojection->projection process at your suggestion. The code compute undistorted images in 24 Hz in 640x480 resolution. Here is what I've done:
json_generate.cpp
img_undistort.cpp
Thank you all very much! Maybe this code can be used in some real-time project. |
Thank you very much for your code could you tell me why i get the error CentralGenericCamera’ was not declared in this scope thankyou |
hi,
I have tried the camera generic model in my project, most of the scenes make sense! But I found that there are some scenes such as the image needs to be resized or distortion correction is need for line detection, I do not know very clearly how to use this camera generic model.
Is there some way to do these things? Thanks very much for your help !
The text was updated successfully, but these errors were encountered: