2009

Maksim Shabunin edited this page Jul 29, 2015 · 1 revision
Clone this wiki locally

OpenCV Meeting notes for 2009 year

2009-01-06

Agenda

  • Update
  • VTK?
  • Finding planes with Radu’s code?

Minutes

  • Russian holidays
    • Vadim working on using Lapack+BLAS in OpenCV
      • Using ATLAS & MKL
  • See about getting some of Radu Ruso’s mesh routines in OpenCV
  • Want to move C-descriptor into OpenCV
    • What’s needed to make a paper out of it
  • 3D reconstruction
    • Good camera
    • Textured light
    • Rotating platform and rotate object captured frame by frame
    • Robot can scan object
  • Structure from motion
    • What is JD working on with 3D reconstruction
  • “OpenCV” entry in manufacturing robots

Action Items

Gary

  • Talk to JD about code
  • Record minutes of tomorrows discussion with the Vision workbench team
  • Talk to Calendar about producing a Star-C-descriptor paper

Vadim

  • Compare OpenCV star detector with ROS star detector

2009-01-13

  • Updates
  • Interface issues (what next)
  • Pattern Recognition
  • 3D functionality

Minutes

Update Gary:

  • Set up stereo capture node and will collect 100’s of outlets and doorknobs
  • Collecting 3D object database of Ikea objects
  • Writing an opencv tutorial for Stanford class, will post that

Update Vadim:

  • Continued work on BLAS and LAPACK interface
    • Eigen vectors and values,
    • SVD,
    • cvsolve,
    • cvinvert and
    • qr decomposition
    • Accuracy is great and performance is great
      • 500×500 matrix is 2.5x faster
      • Small matrixes, 10×10, the new implementation is slower
        • Need to add branches in this case
    • Netlib.org Jack Dungar’s implementation, puts out clapack with same interface
      • Eigen2 lib, small matrices are faster than lapack, but algorithms aren’t as well developed as lapack
  • multiwin replaced using wImage (code from Google)
  • This week is after Russian new year vacation.
    • Anatoli working on Octrees for Spin images
  • CVS becomes obsolete this week

Update Victor

  • Was sick with mono for 3 weeks
  • Outlets
    • Filter keypoints using random forest
      • Train image passes around keypoints
      • 2 class filter for inside/outside outlets, works 90%
      • 70% recognition on hard datasets with textured walls and different lighting condition
      • Low false positive rate
        • Need to turn down false positive rate to gain low false negative (missed detection) rate and then apply stronger models to the detections
      • Needs to wait to final camera setup
    • Need datasets for US outlets for keypoint detection
      • Adapt hole searching code for US outlets
  • Doorknobs
    • Dark round objects using series of thresholds and connected components
    • RCG collected doorknobs together with chessboards.
      • Perspective corrected
      • Trained adaboost
      • Same for test set
        • Results weren’t that good — 40% detection with 15% false alarm
        • Gary suspects making many different templates to match — one each 10 degrees.
  • Random forest improvements
    • Extreme random trees code implemented
      • Bugs fixed while in hospital
      • Compare with same datasets measure test errors.
      • Errors with noise variables
      • Variable importance — correction noise variables
      • If improvements, possible paper

Videri camera is supported in OpenCV

Nearest Neighbor

  • cvFeatureTree, already in OpenCV
  • David Lowes FLANN — test against cvFeatureTree()

Action Items

Gary

  • Get doorknob and outlet datasets to Victor
  • Get outlets from Milestone dataset to Victor
  • Get cannon outlet sites to Argus or Willow sites
  • Get Willow doorknobs to Victor

Vadim

  • Send SVN instructions for Window to Gary

Victor

  • Test random trees against random forest

2009-01-20

Agenda

Update:

  • Milestone2
    • Outlets up close
    • => Victor to Doorknobs?
      • 3D box with chessboard
  • Planning for Summer release, what should be in it

Minutes

Gary
Close range outlet detection to subpixel

Vadim

  1. LAPACK is now included into OpenCV tree.
    cvSVD, cvEigenVV, cvDet, cvSolve and cvInvert have been rewritten using LAPACK;
    cvInvert and cvSolve have got an additional method CV_CHOLESKY, cvSolve also got additional method CV_QR.
  2. Anatoly has implemented a draft version of octree for computing spin images. The existing implementation of octree did not contain the key algorithm for retrieving all the points in the cloud that belong to the given sphere and it was difficult to integrate it, so the new class has been written. Anatoly compared performance of this new code with OBB tree from VTK, which has been used by the existing spin image code and found that the performance of the two is very close now:
  • Hope to speed up spin feature by factor of 10
  • Future to speed up Radu’s histogram of normals

Next release

  • C++ interface with templates
    • 30-50% done
    • Templates will be optional
    • More compatible with STL
  • Much better python interface
    • Mark submitted a patch for python wrapper and use with numpy
  • 3D features/related algorithm
    • Spin images — goal is 10x speedup.
    • Plane fitting from Radu
  • Alpha and/or 4 channel support
  • 2D features
    • C-descriptor (Calandor descriptor)
    • Maybe Boundary fragments
  • Stitching/VSLAM
    • Move in the V-SLAM stitching into OpenCV (?, more probably next …)
    • Place localization/recognition as an app or example code for C-Descriptors
  • Optical flow
    • Dense
    • Rigid
    • Affine
    • Intern from Michael Black
  • Machine learning library
    • Extremely Randomized Trees
    • Nistor Tree
    • Convolutional NN
    • Additions to RF
    • Extend cascade to other features beyond Haar

PRIORITIES

  1. C++ interface
  2. Python
  3. 3D Features (for Obj rec)
  4. 2D Features (for Obj rec)

(no order)

  • MLL
  • Alpha / multi-channel
  • Optical flow improvements
  • Stitching

Victor
Extremely Randomized Trees (ET) algorithm shows comparable performance with published in the original Geurts paper on 4 datasets while on 2 datasets (spambase and vehicle) it demonstrates larger errors (see the table below). We are still investigating this issue, the current main hypothesis is that there is a bug that plays a significant role only on these two datasets. Variable importance charts built for Random Forest (RF) and ET models are similar.

Name RF MLL ET MLL ET Geurts
waveform 17.1 16.06 16.61
vehicle 21.88 47.29 26
spambase 5.41 22.68 4.17
isolet 6.61 10.52 7.61
twonorm7400 4.523944 3.745071 3.53
| ringnorm7400
6.164789 3.138028 3.27

There is a bug on Vehicle and Spambase. UPDATE: This bug is fixed.

The application for automatic tracking of planar objects has been tested on windows and linux with usb camera and video.

A method for learning an RF model of an outlet invariant to affine transformations has been implemented. An RF is learned on patches of frontal view outlets that have been distorted with random affine transforms (following Lepetit’s paper). Each outlet candidate gets a weight (in [0,1]) from this model and low weighted samples are filtered out. Attached is a precision-recall curve of this algorithm obtained by varying the weights threshold. It shows the performance of the model (recall = 1 corresponds to no filtering). The most of false positives are gathered on few images with rich texture.

  • Have outlets detectors in 2D
    • Single image
    • At least 50×50 ROI
    • Based on finding dark round objects (European) and dark round (American)
    • 2 stage RF (European):
      • First stage dark small round holes (where metal prongs goes in)
        • But many false positives, filtered by RF
        • Look for pairs of holes
      • Second stage does the neighborhood of the found holes look like an outlet
    • Need more American data and can train on that
  • Keep going — use this as one method of attack.
  • 3D box relative to chessboard. The RCG (Alexi (tracking, doorknob) Maria (RF, ET))
    • Today he’s training adaboost on doorknobs
    • Error control for planar object position
      • Calibration

Action Items

Victor

  • Send images of current outlet detector
  • Train on Willow images to be sent

Gary

  • Send images of outlets to Victor (today)
  • Try to get several hundred
  • Get doorknob imagery to Victor
  • Take 3D model of doorknob
  • CC Vadim with Ara
  • CC to Jean-Bouguet about 4 channel
  • Talk with Michael about what needs to be done to get C-descriptors in OpenCV
  • Michael Black’s intern
  • Talk with Kurt about moving VSLAM stitching

Vadim

  • Spin images
    • Preliminary study is quite fast, could be optmized more
    • Most of the time in spin image is not in octree, but in the spin image
  • Send me OpenCV goals spreadsheet
  • Put up initial next release goals on OpenCV wiki

2009-01-27

Agenda

  • Updates

Minutes

For OpenCV Release

  • Company Retreat
  • Outlets
    • Bottlenecks
      • Find outlets to sub-mm when robot is “in position”
        • Must detect that nothing is plugged into the outlet
      • Visual servoing after robot touches the wall with the plug
      • Finding the plug orientation
        • Finding a dropped plug(?)
    • Other things which are “good” but not critical for milestone 2
      • Detecting Willow Garage outlets from a distance with and without stereo and/or laser data
      • Detecting outlets in general from a distance with and without stereo or laser data
  • Method for first bottleneck (find plugs to sub-mm accuracy)
  • Using a calibrated camera with rectified images:
    1. Segment the outlet “ovals”, find 4 centroids
    2. Compute homography to canonical view and remap
    3. Match templates to find holes and more exact center of outlets
    4. Re-project more exact center found above into the image to find floating point centers
    5. Use these reprojected floating point centers to again compute a homography to the canonical view and remap
    6. Use templates to find the exact location of outlet holes. This should now be sub-mm precision
    7. Using the homography and measured outlet, compute the X,Y,Z location of the outlet holes
  • Arrange for common repository

Vadim

  • Lapack debugged — now using clapack
    • Simple small matrix operations still use native OpenCV code
    • Try fast lapack algorithms like ATLAS
    • Make use of MKL if available
  • New memory allocator — fast for small matrixies A*B+C
    • ~5x faster than malloc and free
  • Spin images
    • 20x speed up on spin images
    • Compare against Point Feature Histogram PFH
    • Ready in 2-3 more days
    • Can be further optimized by factor of 2x ~ 1 week
    • 20 seconds on 200K points scene of a conference room using 40cm radius
  • Stereo
    • Textured areas have problems, either use textured light or something like Kolmogorov
    • Stereo SSE-2 optimization uses only gcc 4.2 or higher
    • Chessboard detector is breaking on high angle views
      • High angles are not so useful for calibration, but good for plane finding
        • Moderate priority
    • Optimizing Kolmogorov
      • Sparse graph cuts — might not work, but idea is to soft fix points that are unlikely to change
      • Run as background project for now
  • Next week
    • C-descriptor
    • C++ image classes
    • cvBoostPatch() from Steve Gould and from Ian Goodfellow

C descriptor

In …/ros/ros-pkg/visino/calonder_descriptor (it should be named C-descriptor for “Classifier Descriptor”)
I spoke with Michael Calonder about getting his descriptor (which we use in visual odometry and place recognition) into OpenCV. He’s going to write up a short description of how it works. It is already fairly optimized using 16 bit vectors and SSE — it can do a 512×512 image in about 5ms (depending on complexity of course).

They’ve been using the “FAST” detector which has a GPL license. We might consider doing our own FAST detector. But, it can use any intrest point detector, it’s mainly a really good descriptor. At the end of training he used Matlab to compress the vectors using PCA — we’d have to replace that step with clapack.

Place Recognition/Nistor/Kmeans

In ../ros/ros-pkg/vision/place_recognition there are several things of interest:

  1. Improved kmeans code (src/kmeans.cpp)
  2. Nistor vocabulary tree which we can add to MLL (src/vocabulary_tree.cpp)
  3. Place recognition (src/place_recognizer.cpp). This could be sample code or a really useful app that uses (is an example of use of) the C-descriptor above, along with the vocabulary tree and kmeans.

Action Items

Gary

  • Determine time for outlet meetings with Patrick
  • Data with plugs already in outlet
  • Get data to Victor of plug images
  • Put Victor in touch with Michael Calonder
  • Put Vadim in touch with Michael about C-descriptor
  • Find out from Kurt status or priority of stereo speedups
  • Talk to Scott about getting Vadim SSH access to machines in Willow
  • Sandbox in ROS for things like outlet detection experimentation.

Vadim

  • Stereo OpenCV vs Kurt algorithm comparison for speed on a known good machine.

2009-02-03

Agenda

Outlets will be moved to a meeting directly after the OpenCV meeting, so we will no longer discuss outlet specific issues here but instead focus on OpenCV issues.

  • Updates
    • Vadim
    • Victor
    • Gary
  • Progress towards Summer release
    • OpenCV200906
    • Memory limitations in machine learning — design matrix limits
    • Error handling???
  • PnP problem approaches

Minutes

Vadim

  1. Anatoly and me: finished optimization of OctTree & Spinimage code. OpenMP and SSE2 gave ~2x speedup each. In result, the running time of one of our benchmarks dropped from (ROS was 80 secs), new c code is 880ms and now downto ~230ms with SSE and OpenMP. That is, together with ~20x speedup we achieved with new C code, the total speedup is over 75×. The code has been prepared for integration to OpenCV.
  2. The 2 optimizations of the boosting code, proposed by Stephen Gould and Ian Goodfellow, have been integrated to MLL. No feedback from the guys yet.
  3. Studied the C-descriptor source code the relevant papers by Michael Calonder and Vincent Lepetit. The algorithm is pretty clear now and the preparation to integration into OpenCV has been started.
  4. ~30% of the new matrix class integration has been done. This appeared to be significantly bigger task than I expected.
  5. Received the adaptive skin detector code from Farhad Dadgostar; reviewed the code and recieved the corrected variant; The code will be integrated this week.

Victor

  • Outlet detection: calibration, template matching and hole detection have been combined in a single application. The method has been adjusted for another camera (narrow lens resulting in higher resolution outlets). A method for finding a tuple of 4 outlets has been improved for the case of large viewing angle. Recalculation to 3D coordinates in cm scale has been implemented and tested. If an outlet is found correctly the error of hole detection is less than 0.5mm for all cases but one.
  • Extreme trees (Maria): the code has been refactored, all ET functionality has been encapsulated into a separate class inherited from CvRTrees. The performance of RF and ET has been compared. For very deep trees (depth 20) the performance is comparable, while for more shallow trees (less than 10) ET is faster at least 2×.
    • Has natural parallelism, look at it.
    • Parallize all boosted and random prediction trees.
    • Possible paper in statistical convergence to correct answer
  • 3D object tracking (Alexey): the method for tracking 3D will be based on 5 points set by user on two different frames. Stereo pair calibration has been implemented.
    • Right now we are trying to catch a bug in calculating 3D from disparity. Alexi out so this will wait.

Other: Alexey is on vacation Feb2-8, back in the office Feb 9.

Gary

Boosting

  • Will try to see if we’re still artificially limited by memory size

OpenCV errors

  • New implmentation moves completely to exceptions
  • Should work with debugger now

OpenCV website

  • For demos etc

Action Items

NEW

Gary

  • Email Steve Gould about boosting memory limits with Victor and Vadim
  • Email Ian Goodfellow about opencv error catching with Vadim.

Vadim

  • Check ML design matrix

Victor

FROM LAST TIME

Gary

  • Determine time for outlet meetings with Patrick
    • 10am (9pm NN) on Tuesday
  • Data with plugs already in outlet
  • Get data to Victor of plug images
  • Put Victor in touch with Michael Calonder
  • Put Vadim in touch with Michael about C-descriptor
  • (?) Find out from Kurt status or priority of stereo speedups
  • (?) Talk to Scott about getting Vadim SSH access to machines in Willow
    • I talked, but no action yet.
  • Sandbox in ROS for things like outlet detection experimentation.
    • Just use sourceforge and ROS can pull it in if you build say the outlet node.

Vadim

  • Stereo OpenCV vs Kurt algorithm comparison for speed on a known good machine.

2009-02-10

Agenda

Quick updates:

Gary

  • Helped get Mech Turk interface into Willow Garage (thanks to Alexander Sorokin and Scott Hassan’s 2 daze of hacking last week). View some results (click on the numbers under “completed hits”).
    • We can use this for massive data sets and even say sponsor people to create better face recognizers etc.

Vadim

  • Matrix interface
  • 80% of cxcore is ported to the new C++ interface
    • Automatic garbage collection (reference counter based)
    • Can use with STL vector of images and matrixes
    • Have template interface matrix(i,j) does the right thing
      • Thread safety coming, can use atomic operations, efficient on Intel and AMD archs
  • IPP will now be installed/built at configure instead of dynamic
    • Using cmake, if IPP is detected the build flag will be on by default, user can explicitly turn it off (same way as ffmpg libs etc)
  • Decision tree memory optimization, needs about 2 weeks (Maria knows the code a bit, but Vadim expert)
    • Needed to move cascade recognizer to general decision trees, beyond Haar features
    • Think about collaborating with someone to train good working models for the release
  • Spin images are done but waiting to complete the C++ integration before integration (probably by Friday)
  • Focus results, … lower priority

Victor

  • Extreme Trees
    • Categorical support is added
    • Don’t need sorting of variables, so lower memory requirements
    • Can handle 100K samples of 1000 features, Extreme trees worked OK
    • Code is in SVN
    • Recognition performance is comparable to RF but faster
      • Inherently more parallel
      • Try on outlets?
  • 3D object tracking (Alexander was on vacation)
    • Estimate pose of box in space from motion and knowlege of box
    • Function for disparity to 3D has bug
    • 2D version … finished … but special case of 3D, integrate

OpenCV release tracking

Minutes

Tracking the summer release:

  1. General Structure:
    • Decisions on library structure (Google’s request, separate packages as is …)
      • Provide way to build OpenCV statically, then size varies with binary size
  2. New Features
    • C++ interface
      • … see above
    • Better python
      • Vadim, Alex
      • C++ will take another month but can improve python in parallel
    • 2D features C-descriptor, HoG Boundary Fragments …
      • C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
      • Boundary fragments … working on focus.
      • Get contributions of 2D features externally
      • MSER … (Gary has code)
      • Star is already in and in for ROS build
    • Alpha channel, alternatives to contours RLE
      • Introduce functions with advance ways of blending
      • Mask channels
      • Optical flow
    • 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation
      • SPIN almost done
      • Radu is organizing his code into a library … maybe we can just lift this
    • VSLAM
      • Wait on what Kurt and James get together
    • Image stitching
      • will ask Jean-Yves for pointers
    • Optical flow
      • Move code derived form former work in OpenCV
    • MLL
      • Extreme trees are coming
      • Reduce memory footprint
      • Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
      • Regression tests … maybe general cross validation
    • Misc: Focus detector, adaptive skin tracking
      • As discussed above
  3. Optimizations
    • .
  4. Documenation
    • .
  5. High GUI
    • .

Action Items

NEW

Meeting next week will be on Wednesday

Gary

  • Send Vadim MSER code
  • Ask Jean-Yves for priority list of functions to work with mask and alpha channels
  • Ask Jean-Yves about image stitching support/libraries/functionality
  • Ask Radu about his library progress.
  • Send email reminder for Wednesday meeting next week.

Vadim

  • Adaptive skin tracker app/test
  • Finish cxcore rewrite, make sure all the tests pass, update the documentation.
  • Start cv module revision.
  • Integrate spin images.

Victor

FROM LAST TIME

Gary

  • Email Steve Gould about boosting memory limits with Victor and Vadim
  • Email Ian Goodfellow about opencv error catching with Vadim.

Vadim

  • Check ML design matrix

Victor

2009-02-18

Agenda

Accomplishment Tracking

  • .

Chessboard issues

  • Detection failures
  • 2 functions? One chessboards for calibration, another chessboards for pose?

3D capture, discuss implementation of

  • http://www.robots.ox.ac.uk/~awf/turntable/index.html

cxcore re-write, cv-rewrite

Spin, skin

Release tracking.

Minutes

Tracking the summer release:

  1. General Structure:
    • Decisions on library structure (Google’s request, separate packages as is …)
      • Provide way to build OpenCV statically, then size varies with binary size
  2. New Features
    • C++ interface
      • … see below
    • Better python
      • Vadim, Alex
      • C++ will take another month but can improve python in parallel
    • 2D features C-descriptor, HoG Boundary Fragments …
      • C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
      • Boundary fragments … working on focus.
      • Get contributions of 2D features externally
      • MSER … (Gary has code)
      • Star is already in and in for ROS build
    • Alpha channel, alternatives to contours RLE
      • Introduce functions with advance ways of blending
      • Mask channels
      • Optical flow
    • 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation
      • SPIN almost done
      • Radu is organizing his code into a library … maybe we can just lift this
    • VSLAM
      • Wait on what Kurt and James get together
    • Image stitching
      • will ask Jean-Yves for pointers
    • Optical flow
      • Move code derived form former work in OpenCV
    • MLL
      • Extreme trees are coming
      • Reduce memory footprint
      • Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
      • Regression tests … maybe general cross validation
    • Misc: Focus detector, adaptive skin tracking
      • As discussed above
  3. Optimizations
    • .
  4. Documentation
    • .
  5. High GUI
    • .

New Interface

  • cxcore revision is complete (except for XML/YAML input/output functions). All the algorithmic tests for cxcore and cv pass, all the samples run well.
    • As a small illustration of the new API capabilities, here is a some code that I used to debug SVD:

// matrix A comes in
SVD svd(A);
int n = A.rows;

double a = norm(svd.u*svd.u.t() - Mat::eye(n,n), CV_C);
double b = norm(svd.vt*svd.vt.t() - Mat::eye(n,n), CV_C);
int nz = countNonZero(svd.w >= 0);
double c = norm(svd.u*Mat::diag(svd.w)*svd.vt - Mat::eye(n,n), CV_C);
CV_Assert( a < 1e-5 && b < 1e-5 && c < 1e-5 && nz == n );
  • Can template this
    • Can create image with up to 64 channels
    • Some functions limit to 4 channels
    • Splitting channels is easy
      • Work with IPP, for cache, want to extract channel

General Issues:

Chessboard

  • cvFindCornersSubpix fails with large search window, diverges.
    • Ordering of chessboard corners and splitting
    • Will lookat
  • Initial location of corners can be wrong
    • Extreme angles can create problems for morphological operations
      • vertical and horizontal operations can make distance between corners smaller than within squares

3D

  • http://www.robots.ox.ac.uk/~awf/turntable/index.html
    Skin detection should work using highGUI camera

Spin images, trying on Ikea

3D tracking (Victor, Alexi) (3D box)

  • 3d positions of markers using Chessboard
    • Currently unstable depending on where the user clicks
    • Somewhere in disparity to 3D transformation
    • Generating tests on synthetic data by tomorrow.

Memory use with MLL

  • Shrank memory use several times
  • Debugging

Action Items

NEW

ALL

  • Send in list of what’s been accomplished so far in OpenCV since Willow started supporting development.

Gary

  • Look into sending prosilica and other cameras to Argus
  • Send full data link with mask images for Spin image work
  • Get Zisserman 3D paper to Vadim (on ACM)

Vadim

  • Look at better calibration and pose object
    • Equal or better in accuracy and stability
  • Check what you think 3D scanning would take

Victor

FROM LAST TIME

Meeting next week will be on Wednesday

Gary

  • Send Vadim MSER code
  • [but he didn’t answer yet] Ask Jean-Yves for priority list of functions to work with mask and alpha channels
  • [same] Ask Jean-Yves about image stitching support/libraries/functionality
  • Ask Radu about his library progress. Radu:

    We took a few steps today to unify the point cloud -related libraries in ROS under one umbrella. The result is that
    there is now a point_cloud_mapping package which contains the following things:
    - cloud_geometry: areas, distances, intersections, 3D normals/features estimation, norms, projections, statistics,
    transformations, etc;
    - cloud_io: a library for Input/Output PCD (Point Cloud Data) processing;
    - cloud_kdtree: a library for nD spatial decomposition and nearest neighbors searches for PointCloud messages using k-D
    trees;
    - cloud_octree: a library for 3D spatial decomposition of PointCloud messages using octree data structures;
    - sample_consensus: a library for SAmple Consensus (SAC) like methods (RANSAC, MSAC, MLESAC, etc,) and geometric shape
    models;
    There are also 2 basic nodes included with the package:
    - cloud_downsampler_node: a node which uniformly downsamples a 3D point cloud
    - normal_estimation_node: a node which estimates local surface properties at each point such as surface normals,
    curvature changes, moment invariants, etc.
  • Send email reminder for Wednesday meeting next week.

Vadim

  • Adaptive skin tracker app/test
  • Finish cxcore rewrite, make sure all the tests pass, update the documentation.
  • Start cv module revision.
  • Integrate spin images.

Victor

2009-02-24

Agenda

  • Accomplishments
  • Interface update
  • Chessboard improvements
  • 3D Box relative to chessboard
  • 3D Object Capture paper/analysis
  • Spin images
  • Summer release tracking

Minutes

Tracking the summer release:

  1. General Structure:
    • Decisions on library structure (Google’s request, separate packages as is …)
      • Provide way to build OpenCV statically, then size varies with binary size
  2. New Features
    • C++ interface
      • cxcore
      • cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
    • Better python
      • Vadim, Alex
      • C++ will take another month but can improve python in parallel
    • 2D features C-descriptor, HoG Boundary Fragments …
      • C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
        • No re-implementation yet. Code based on randomized tree, but specialized version.
      • Boundary fragments … working on focus.
      • Get contributions of 2D features externally
      • MSER … (Gary has code)
      • HoG
      • Star is already in and in for ROS build
    • Alpha channel, alternatives to contours RLE
      • Introduce functions with advance ways of blending
      • Mask channels
      • Optical flow
    • 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation
      • SPIN almost done
      • Radu is organizing his code into a library … maybe we can just lift this
    • VSLAM
      • Wait on what Kurt and James get together
    • Image stitching
      • will ask Jean-Yves for pointers
    • Optical flow
      • Move code derived form former work in OpenCV
    • MLL
      • Extreme trees are coming
      • Reduce memory footprint
      • Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
      • Regression tests … maybe general cross validation
      • Generalize cascade
    • Misc: Focus detector, adaptive skin tracking
      • As discussed above
  3. Optimizations
    • .
  4. Documentation
    • .
  5. High GUI
    • .

OTHER

New Interface

  • Whole new interface for cxcore is finsished
    • Moving to cv
    • Examples: Update samples. .c code will be old interface, .cpp will be new interface
    • Output is simplified by automatically correct type
    • Start effort to doxygen (lower background priority)

Chessboard code that

  • Vadim has released a patch that seems to fix all known problems. Congrats!
  • High angle (60 degrees) boards still cannot be done
  • Detect lines, automatically detect board size, perhaps remap image to frontal view.
    • Sub-pixel finding is now put inside the algorithm rather than after.

3D box

  • 3D object tracking: the code is in debugging phase. The calibration of a stereo pair is still unstable, producing large reprojection errors too often.
  • Alexi not much time to work on in
  • Make relative to chessboard

3D Object Capture

  • Bundle adjustment missing from OpenCV
  • Mesh reconstruction, not in OpenCV
  • Might take a month if we can’t get the above code
    • http://www.3dsom.com/index.html seems to use the same technique.

Spin images

  • Not incorporated yet, code is ready, just time to integrate
    • Test code is ready
    • 1 or 2 days

Extremely random trees

  • Put a callback in the splitting function

MSER

  • Alex has MSER code

Action Items

NEW

Gary

  • Ask Kurt about where Bundle adjustment is … if it is
  • Send Victor Florian’s paper

Vadim

  • Doxygen documentation as background, lower priority task.

Victor

FROM LAST TIME

ALL

  • Send in list of what’s been accomplished so far in OpenCV since Willow started supporting development.

Gary

  • Look into sending prosilica and other cameras to Argus
  • Send full data link with mask images for Spin image work
  • Get Zisserman 3D paper to Vadim (on ACM)

Alexi

  • Finish up 3D box

Vadim

  • Look at better calibration and pose object
    • Equal or better in accuracy and stability
  • Check what you think 3D scanning would take

Victor

2009-02-24

Agenda

  • Accomplishments
  • Interface update
  • Chessboard improvements
  • 3D Box relative to chessboard
  • 3D Object Capture paper/analysis
  • Spin images
  • Summer release tracking

Minutes

Tracking the summer release:

  1. General Structure:
    • Decisions on library structure (Google’s request, separate packages as is …)
      • Provide way to build OpenCV statically, then size varies with binary size
  2. New Features
    • C++ interface
      • cxcore
      • cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
    • Better python
      • Vadim, Alex
      • C++ will take another month but can improve python in parallel
    • 2D features C-descriptor, HoG Boundary Fragments …
      • C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
        • No re-implementation yet. Code based on randomized tree, but specialized version.
      • Boundary fragments … working on focus.
      • Get contributions of 2D features externally
      • MSER … (Gary has code)
      • HoG
      • Star is already in and in for ROS build
    • Alpha channel, alternatives to contours RLE
      • Introduce functions with advance ways of blending
      • Mask channels
      • Optical flow
    • 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation
      • SPIN almost done
      • Radu is organizing his code into a library … maybe we can just lift this
    • VSLAM
      • Wait on what Kurt and James get together
    • Image stitching
      • will ask Jean-Yves for pointers
    • Optical flow
      • Move code derived form former work in OpenCV
    • MLL
      • Extreme trees are coming
      • Reduce memory footprint
      • Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
      • Regression tests … maybe general cross validation
      • Generalize cascade
    • Misc: Focus detector, adaptive skin tracking
      • As discussed above
  3. Optimizations
    • .
  4. Documentation
    • .
  5. High GUI
    • .

OTHER

New Interface

  • Whole new interface for cxcore is finsished
    • Moving to cv
    • Examples: Update samples. .c code will be old interface, .cpp will be new interface
    • Output is simplified by automatically correct type
    • Start effort to doxygen (lower background priority)

Chessboard code that

  • Vadim has released a patch that seems to fix all known problems. Congrats!
  • High angle (60 degrees) boards still cannot be done
  • Detect lines, automatically detect board size, perhaps remap image to frontal view.
    • Sub-pixel finding is now put inside the algorithm rather than after.

3D box

  • 3D object tracking: the code is in debugging phase. The calibration of a stereo pair is still unstable, producing large reprojection errors too often.
  • Alexi not much time to work on in
  • Make relative to chessboard

3D Object Capture

  • Bundle adjustment missing from OpenCV
  • Mesh reconstruction, not in OpenCV
  • Might take a month if we can’t get the above code
    • http://www.3dsom.com/index.html seems to use the same technique.

Spin images

  • Not incorporated yet, code is ready, just time to integrate
    • Test code is ready
    • 1 or 2 days

Extremely random trees

  • Put a callback in the splitting function

MSER

  • Alex has MSER code

Action Items

NEW

Gary

  • Ask Kurt about where Bundle adjustment is … if it is
  • Send Victor Florian’s paper

Vadim

  • Doxygen documentation as background, lower priority task.

Victor

FROM LAST TIME

ALL

  • Send in list of what’s been accomplished so far in OpenCV since Willow started supporting development.

Gary

  • Look into sending prosilica and other cameras to Argus
  • Send full data link with mask images for Spin image work
  • Get Zisserman 3D paper to Vadim (on ACM)

Alexi

  • Finish up 3D box

Vadim

  • Look at better calibration and pose object
    • Equal or better in accuracy and stability
  • Check what you think 3D scanning would take

Victor

2009-03-03

Agenda

  • SURF and OpenCV Announce
  • Marius Muja is at Willow for 6 months
    • FLANN incorporation
  • New interface update
  • 3D Box — it slips
  • 3D Model
  • MSER

Minutes

Tracking the summer release:

  1. General Structure:
    • Decisions on library structure (Google’s request, separate packages as is …)
      • Provide way to build OpenCV statically, then size varies with binary size
  2. New Features
    • C++ interface
      • cxcore
      • cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
    • Better python
      • Vadim, Alex
      • C++ will take another month but can improve python in parallel
    • 2D features C-descriptor, HoG Boundary Fragments …
      • C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
        • No re-implementation yet. Code based on randomized tree, but specialized version.
      • Boundary fragments … working on focus.
      • Get contributions of 2D features externally
      • MSER … (Gary has code)
      • HoG
      • Star is already in and in for ROS build
    • Alpha channel, alternatives to contours RLE
      • New alpha transparency (per pixel) added
      • Introduce functions with advance ways of blending
      • Mask channels
      • Optical flow
    • 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation
      • SPIN almost done
      • Radu is organizing his code into a library … maybe we can just lift this
    • 3D capture http://www.3dsom.com/index.html
    • VSLAM
      • Wait on what Kurt and James get together
    • Image stitching
      • will ask Jean-Yves for pointers
    • Optical flow
      • Move code derived form former work in OpenCV
    • MLL
      • Extreme trees are coming
      • Reduce memory footprint
      • Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
      • Regression tests … maybe general cross validation
      • Generalize cascade
    • Misc: Focus detector, adaptive skin tracking
      • As discussed above
  3. Optimizations
    • .
  4. Documentation
    • .
  5. High GUI
    • .

OTHER

OpenCV Announce …

  • SURF is in #1 in change log
  • Post change log to wiki
  • OpenCV Weekly …

FLANN

  • Some timing results at bottom
  • Code is in good shape
    • 3D party directoy can be put there
    • For mac, can be put in system path
    • Linux can select
  • In CV module, can create a wrapper for CvMat interface to FLANN
    • Make test, so can use included version of FLANN
    • First looks to internal unless user specifies external
      • Put flag to download external
      • We keep reasonably up to date with external

Interface update

  • cxcore in SVN
    • Mac and Windows,
    • Linux: gcc caused changes due to template handling differences
  • convolution in cv
    • Then geometrical transforms
    • Optical flow
    • Add new alpha blending function with separate transparency for each pixel
    • Further functions can be lower priority

3D box

  • Box slips up (pretty sure)
  • Image is rectified
    • If camera is close, distortion can be large
    • Tomorrow will ship new video with box being tracked
  • Could be chessboard slippage ?

3D Object Capture

  • Bundle adjustment missing from OpenCV
  • Mesh reconstruction, not in OpenCV
  • Might take a month if we can’t get the above code
    • http://www.3dsom.com/index.html seems to use the same technique.
  • Initialization of extrinsics (pose) is hardest part of the code
  • Different from general SFM or stereo
  • smart inversion of J^t*J
    • pass to SVD (might be 10x slower but easier)

MSER

  • Gary sent
    • Victor looking at right now,
      • Alex doing his own version faster at lower priority ~ couple of weeks

SURF

  • detection and descriptor united
    • Should be split
    • “Find_obj.cpp” is demo code in samples/c

Vadim:

  1. cxcore has been built with GCC 4.0 and GCC 4.2 on MacOSX. Quite a few changes have been made because GCC handles some template stuff differently and usually catches more bugs. Another portion of changes were made to make all the tests pass. This tested version of cxcore has been committed to SVN repository.
  2. The work on revising cv has been started, new interface for filtering functions (linear separable and non-separable, Laplacian, median, morphogical operations) is ~90% ready (the code was written and now debugged).
  3. fixed 1 minor bug in boosting (reported by Ian Goodfellow) and 2 bugs in camera calibration (reported by Sergey).
  4. Anatoly started the work on benchmarking various fast n-dim search algorithms for quality & speed. For now we have 3 algorithms in OpenCV: kdtree, spilltree, LSH (locally-sensitive hash), and there is also FLANN. We could not run LSH properly yet, but here are some first results on synthetic uniformly-distributed data in n-dim cubes:
    • tAvgDist is the average (over the test set) distance to the nearest point, as returned by various alorithms (DumbBruteForce is the naive algorithm), the closer to the naive algorithm, the better.
    • M1 is average over the test set of (min_dist/average_dist), the smaller the better.
    • M2 is the number of times the returned point was closer than MAX, 0)
    • and M3 is average of (average_dist – min_dist)/stddev(distance). More than 3 should good enough from statistical point of view, but the larger – the better.
  5. These preliminary results show that kd-tree is the fastest method and generally not that bad, FLANN is as accurate as naive algorithm, yet significantly faster (but significantly slower than kd-tree, we wonder, whether it can be tuned to be less accurate but faster)
  6. spill tree is only good for low-dimensional space, otherwise brute force is much faster.
  7. LSH is yet to be tested.

Action Items

NEW

Gary

  • Make change log wiki page, and send to Vadim
  • Make OpenCV Montly Newsletter and send to Vadim
  • Ask J-Y again about mask image uses/image stitching
  • Find JD’s code and send to Vadim
  • Send email to Radu to point to mesh code/techniques
  • Calendor to Andrew Ng

Vadim

  • Put up change log wiki page and monthly page after Gary set’s up

Victor

  • Do new video of box

FROM LAST TIME

Gary

  • Ask Kurt about where Bundle adjustment is … if it is
    • Umm, JD Chen (no longer with Willow) supposedly did this, but for stereo code, sent him an email
  • Send Victor Florian’s paper
    • http://www.robots.ox.ac.uk/~vgg/publications/papers/schroff08.pdf
    • http://www.robots.ox.ac.uk/~vgg/publications/papers/schroff06.pdf

Vadim

  • Doxygen documentation as background, lower priority task.

Victor

2009-03-10

Agenda

Summer of Code

New interface status

FLANN Benchmarks

Status of Extremely Random Trees — is it in

MSER?

3D Box

3D capture

  • Bundle adjustment missing from OpenCV
  • Mesh reconstruction, not in OpenCV
  • Might take a month if we can’t get the above code
    • http://www.3dsom.com/index.html seems to use the same technique.

Minutes

Tracking the summer release:

  1. General Structure:
    • Decisions on library structure (Google’s request, separate packages as is …)
      • Provide way to build OpenCV statically, then size varies with binary size
  2. New Features
    • C++ interface
      • cxcore
      • cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
    • Better python
      • Vadim, Alex
      • C++ will take another month but can improve python in parallel
    • 2D features C-descriptor, HoG Boundary Fragments …
      • C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
        • No re-implementation yet. Code based on randomized tree, but specialized version.
      • Boundary fragments … working on focus.
      • Get contributions of 2D features externally
      • MSER … (Gary has code)
      • HoG
      • Star is already in and in for ROS build
    • Alpha channel, alternatives to contours RLE
      • New alpha transparency (per pixel) added
      • Introduce functions with advance ways of blending
      • Mask channels
      • Optical flow
    • 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation
      • SPIN almost done
      • Radu is organizing his code into a library … maybe we can just lift this
    • 3D capture http://www.3dsom.com/index.html
    • VSLAM
      • Wait on what Kurt and James get together
    • Image stitching
      • will ask Jean-Yves for pointers
    • Optical flow
      • Move code derived form former work in OpenCV
    • MLL
      • Extreme trees are coming
      • Reduce memory footprint
      • Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
      • Regression tests … maybe general cross validation
    • Misc: Focus detector, adaptive skin tracking
      • As discussed above
  3. Optimizations
    • .
  4. Documentation
    • .
  5. High GUI
    • .

OTHER

Summer of code

  • We already have not just a cascade, but a cascade-tree
  • Gradient boosting trees(?) — project?
  • Optimize general boosting, not haar
  • Have Mechanical Turk
  • Image stitching
  • HighGUI

New interface

  • Filtering functions in cv all pass test
    • SSE optimizations added for most important filtering functions
  • Image warping next
  • Usergroup glitch (old macro) posted to SVM

FLANN

  • Investigation is continuing by Anatoly
    • With new FLANN #2, the performance gap is smaller
      • Accuracy is the same
      • kd-tree in OpenCV is very fast, but not so accurate
      • FLANN, accuracy is high, but speed is slower.
        • We made it faster, with same accuracy, but don’t know how to make it less accurate for more speed.

Extremely Random Trees

  • Is in. Needs sample code

MSER

  • Got it, stripped out other parts and compiled it.
  • Alexander working on it.

3D Box

  • There was a bug, found it, corrected it.
  • Sent email

REPORTS

Vadim

  1. Debugged the new filtering functions (called by cvSmooth, cvFilter2D, cvErode, cvDilate, cvSobel). Now all the tests pass. Also, added SSE2 optimization to some important functions, like morphology and separable filters (like Sobel and Gaussian), so they should be much faster even if IPP is not available.
  2. Started revision of image warping functions.
  3. Anatoly continued investigation of various n-dim search algorithms. In the latest version of benchmark and with FLANN #2 the FLANN part runs faster, but still there is gap in speed-accuracy tradeoff. Also, the LSH part still does not give adequate results to be included into the benchmark. These 2 items are being investigated. Anatoly also ported the benchmark to Linux and sent it to Marius.

Victor

  • See outlet detection for main report
  • 3D tracking: several bugs have been corrected, the new video is at ftp://argus-cv.dnsalias.org/3DTrackDemo/logitec_demo.avi, login willow, password awgarage.

Action Items

NEW

Gary

  • Put up GSofC projects
  • Job application page
  • Put up application
  • Ask Marius to interact with Anatoly on FLANN

Vadim

  • Put up GSofC projects

Victor

  • Put up GSofC projects

FROM LAST TIME

Gary

  • Make change log wiki page, and send to Vadim
  • Make OpenCV Montly Newsletter and send to Vadim
  • Ask J-Y again about mask image uses/image stitching
  • Find JD’s code and send to Vadim
  • Send email to Radu to point to mesh code/techniques
  • Calendor to Andrew Ng

Vadim

  • Put up change log wiki page and monthly page after Gary set’s up

Victor

  • Do new video of box

2009-03-17

Agenda

[No meeting next week, I won’t be in]

Google Summer of Code

HOG and SPIN

3D rotation + Translation of stereo images

Updates

  • New interface

Monthly announce, Change Logs

FLANN Benchmarks

3D Box

3D capture

Minutes

OpenCV Summer Release

  1. General Structure:
    • Decisions on library structure (Google’s request, separate packages as is …)
      • Provide way to build OpenCV statically, then size varies with binary size
  2. New Features
    • C++ interface
      • cxcore
      • cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
    • Better python
      • Vadim, Alex
      • C++ will take another month but can improve python in parallel
    • 2D features C-descriptor, HoG Boundary Fragments …
      • C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
        • No re-implementation yet. Code based on randomized tree, but specialized version.
      • Boundary fragments … working on focus.
      • Get contributions of 2D features externally
      • MSER … (Gary has code)
      • HoG
      • Star is already in and in for ROS build
    • Alpha channel, alternatives to contours RLE
      • New alpha transparency (per pixel) added
      • Introduce functions with advance ways of blending
      • Mask channels
      • Optical flow
    • 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation
      • SPIN almost done
      • Radu is organizing his code into a library … maybe we can just lift this
    • 3D capture http://www.3dsom.com/index.html
    • VSLAM
      • Wait on what Kurt and James get together
    • Image stitching
      • will ask Jean-Yves for pointers
    • Optical flow
      • Move code derived form former work in OpenCV
    • MLL
      • Extreme trees are coming
      • Reduce memory footprint
      • Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
      • Regression tests … maybe general cross validation
    • Misc: Focus detector, adaptive skin tracking
      • As discussed above
  3. Optimizations
    • .
  4. Documentation
    • .
  5. High GUI
    • .

OTHER

GSoC

  • Already received an application, but Google decides who’s in on the 18th
  • I’ve been working on some stereo data collection for people detection
    • Also for object detection too

HOG

  • 2D feature

SPIN

  • Code and
  • Demo reads vrml
  • Constructs oct tree C++
  • Has compute normal points separate functions (takes oct tree)
  • Then computes spin image

3D ROTATION TO NEW IMAGE

  • 3d perspective transform
  • perspective projection
  • for each point, find generalized interpolation
    • Computer weighted sum of K nearest points
    • To search in projected view, use quad tree or regular grid

Vadim update

  1. Made another few corrections in filtering functions to make the API a bit more convenient. Updated filtering-depdendent functions (that used not cvSobel, cvFilter2D etc., but the internal API): SURF, Snakes, CornerMinEigenVal etc. Now the tests for these functions pass too. Here are some performance numbers (on Core2Duo #6GHz laptop, assuming that performance of C code has not been changed much comparing to the old version of OpenCV, this is how much faster the summer version will be comparing to #0 and #1pre).
  2. Started the work on updating image warping functions. cvResize & cvWarpAffine have been rewritten and optimized using SSE2 intrinsics. The code is now being debugged.

FLANN:

Anatoly finished benchmarking FLANN and OpenCV’s KD-tree.

  • Below is the table (M1 and M3 mean the same thing as in previous report:
    • M1 – ratio of the distance to the found neighbor to the average distance to points in DB. The smaller – the better.
    • M3 – normalized to standard deviation difference between average distance and best distance. >= 2-2.5 is already very good. The larger – the better.
  • Conclusion
    • Current conclusion is that OpenCV’s kd-tree is comparable to FLANN in both speed and accuracy (by varying parameters we can get either better accuracy or better performance).
    • Anatoly now started working on HoG and the framework for model matching using spin-images.

3D Box

  • Sent video of tracking another box
  • Put in local SVN

Application area

  • Under trunk put OpenCV Apps

3D Reconstruction

  • Nothing

Action Items

NEW

Gary

  • Talk to Radu about 3D rotation of points and create an image
  • Gary to send Anatoly stereo data for trying HOG and SPIN
  • Send gaussian curvature paper
  • Kurt to send paper on fast plane fit
  • Histogram of Gradient link

Vadim

  • Anatoly — prioritize HOG, SPIN
    • Fast plane fit

Victor

  • Set up the first OpenCV appliation: 3D box
    • Video of it working
    • Raw video that spits out marked object

FROM LAST TIME

Gary

  • Put up GSofC projects
  • Job application page
  • Put up application
  • Ask Marius to interact with Anatoly on FLANN

Vadim

  • Put up GSofC projects

Victor

  • Put up GSofC projects

2009-03-31

Agenda

Interface progress

  • Python machine learning

Updates for

Spin and HOG progress

  • How was HOG done?

Trying 3D capture code

Point cloud library

Application areas

New HighGUI

Improving the machine learning interface(?)

  • There is also the issue of a new classifier design for robotics that needs the following capabilities:
    • Needs to do one shot learning (but can refine later)
    • Needs to be able to combine separately learned objects in a sub-linear way
    • Needs the ability to express some structure in the data, particularly spatial or geometric structure.

Minutes

OpenCV Summer Release

  1. General Structure:
    • Decisions on library structure (Google’s request, separate packages as is …)
      • Provide way to build OpenCV statically, then size varies with binary size
  2. New Features
    • C++ interface
      • cxcore
      • cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
    • Better python
      • Vadim, Alex
    • 2D features C-descriptor, HoG Boundary Fragments …
      • C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
        • No re-implementation yet. Code based on randomized tree, but specialized version.
      • Boundary fragments … working on focus.
      • Get contributions of 2D features externally
      • MSER … (Gary has code)
      • HoG
      • Star is already in and in for ROS build
    • Alpha channel, alternatives to contours RLE
      • New alpha transparency (per pixel) added
      • Introduce functions with advance ways of blending
      • Mask channels
      • Optical flow
    • 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation
      • SPIN almost done
      • Radu is organizing his code into a library … maybe we can just lift this
    • 3D capture http://www.3dsom.com/index.html
    • VSLAM
      • Wait on what Kurt and James get together
    • Image stitching
      • will ask Jean-Yves for pointers
    • Optical flow
      • Move code derived form former work in OpenCV
    • MLL
      • Extreme trees are coming
      • Reduce memory footprint
      • Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
      • Regression tests … maybe general cross validation
    • Misc: Focus detector, adaptive skin tracking
      • As discussed above
  3. Optimizations
    • .
  4. Documentation
    • .
  5. High GUI
    • .

OTHER

Vadim:

  1. Revised, converted to the new API and tested:
    • all the image warping functions (Resize, WarpAffine, WarpPerspective, Remap & Undistort), with SSE2 optimization the performance improves up to 50-100%.
    • integral images
    • fixed-level threshold and adaptive threshold + SSE2 optimization
    • functions for background statistics accumulation
    • For the previously added median blur function added SSE2 optimization for 3×3 and 5×5 kernels, up to x10 speedup.
  2. Fixed haartraining app. problem (reported by Caroline Pantofaru) by reverting couple of files to the versions from OpenCV #0.
  3. Modified spin images to use the new C++ matrix class. The updated code is in SVN.
  4. Integrated MSER code by Liu Liu. For now the C interface is retained.
  5. Anatoly implemented function for extracting HoG features based on the contributed by Alberto Orlandi code. At new version is also ~50% faster. Some further optimization is possible. The code is in SVN now.
  6. Started working with the dataset of IKEA objects. HoG features did not provide any reasonable output (e.g. comparing to SURF), because the object outlines are very simple and HoG seems to require a classifier on top of it. Now trying to run spin images.
  • Next for API is optical flow
    • Pyramid and adding
    • Dense optical flow)
    • High Gui
  • MLL
    • Extremely random trees Importance Variables
    • Input interface: to csv and loaded with standard function in SVN tomorrow
  • ROS stable this week.
    • latest_tested_snapshot is the SVN flag for ROS

Stereo data

  • On mechanical turk
  • Radu, has developed a plane finder interface

Nearest neighbor

  • .

Spin features

  • Tried on synthetic data and vmrl files
  • Missing elements vs Andrew Johnson features
    • Using geometric constraints if occluded object
  • API 3D point cloud from oct tree
    • Pass set of points into spin function
    • List of 3D locations to compute spin images at
    • Computes normals
    • Pass pts and normals to a function that computes spin images
      • Support is the radius of included points
  • Memory and time hogs — hard to optimize

HOG

  • Anatoli tried HOG in a “Haar style” manner
  • HOG features compared in matching set of points

Lighting

  • Want to start getting in specularity and lighting direction detectors
  • Train by by rendering

Stanford feature

  • Sub patches from mean

3D capture code

  • Start Anatoly on 3D capture

Action Items

NEW

Gary

Vadim

  • Check kdd trees on small vectors

Victor

FROM LAST TIME

Gary

  • Talk to Radu about 3D rotation of points and create an image
  • Gary to send Anatoly stereo data for trying HOG and SPIN
    • Look in
    • http://vault.willowgarage.com/wgdata1/vol1/IKea_Related/Class
    • For file Objects_kjlKJL.tar.gz (0.5G)
  • Send gaussian curvature paper
  • Kurt to send paper on fast plane fit
  • Histogram of Gradient link

Vadim

  • Anatoly — prioritize HOG, SPIN
    • Fast plane fit

Victor

  • Set up the first OpenCV appliation: 3D box
    • Video of it working
    • Raw video that spits out marked object

2009-04-07

Agenda

  • We need to update:
    • Accomplishments
    • Monthly announce
    • Change Logs Can we get new functions listed here?
    • Contributors credit page?
    • Update what people are working on.
  • Vadim
    • Stereo block match at two scales. Progress?
    • kd_tree tests for small vectors?
    • New interface progress
    • 3D Capture — software in? Testing?
    • Python interface based on numpy matrices?
    • Optical flow code?
  • Victor
    • Code for 3D box tracking
    • Chamfer matching
      • 2D
      • 3D?
    • Machine learning reading csv
      • Other MLL improvements

Minutes

Victor

  • 3D box code (put into OpenCV application section)
  • Chamfer matching
    • Used in outlet matching — works well for orange matching
      • Many false positives in high texture
      • Works worse on white on white because edges are unreliable
      • Useful for European outlets
      • Edge template better, but depends on edge fidelity
  • MLL
    • a major commit to opencv svn.
      • The new memory-efficient structure for tree-based classifiers has been implemented. The new memory limitations are still under investigation (because of the recent changes in cxcore).
      • About 2X saving for old version of OpenCV, looking at new version
      • 10,000 samples, 1000 variables.
      • Doing new tests (Maria)

Application section

  • Same svn link, but put opencv_extra
  • For example https://opencvlibrary.svn.sourceforge.net/svnroot/opencvlibrary/trunk/opencv_extra/3d/createsamples3d/

Vadim

  • Stereo starting tommorrow
  • kd-tree on small vectors — will send table

Activity results:

  1. Converted to the new API and optimized (by factor of 2) pyramid functions.
  2. Integrated Fuzzy MeanShift Tracker by Farhad Dadgostar (the same person who contributed adaptive skin detector).
  3. Integrated Farnerback dense optical flow.
  4. Anatoly started investigation of 3DSom software for 3D reconstruction. A brief report has been sent.

Plans for this week:

  1. Purchase some lighting setup for better background isolation. We have found some good possible setups in the internet already.
  2. Finish optical flow intergration (other dense optical flow algorithm and better sparse optical flow).
  3. Implement adaptive-window variant of stereo correspondence algorithm
  4. Issues: we still have not received the license update from 3DSom.

Python

  • Can create numpy array and call OpenCV function
  • C-type tool in more use
  • Supports overload

Optical Flow

  • Forward back algorithm
  • Have another dense algorithm almost ready, will integrate
  • Improve sparse optical flow with color and image gradients
  • About 1 week more work (delayed for stereo)

Contours and Object detection

  • Need C++ interface
  • Contours
    • 2 week working time (data structure issues)
  • Object detection
    • Probably 1 week

HighGUI and MLL

  • Can be converted in interface
  • HighGUI fast
  • MLL ~ 1 month

External code, improvement in

The details of the implementation can be found here:

http://www.springerlink.com/content/p32p311820r63k68/

and one of the related videos also is here:

http://www.youtube.com/watch?v=gVPCrQBKKqE&feature=channel_page

OpenCV Summer Release

  1. General Structure:
    • Decisions on library structure (Google’s request, separate packages as is …)
      • Provide way to build OpenCV statically, then size varies with binary size
  2. New Features
    • C++ interface
      • cxcore
      • cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
    • Better python
      • Vadim, Alex
    • 2D features C-descriptor, HoG Boundary Fragments …
      • C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
        • No re-implementation yet. Code based on randomized tree, but specialized version.
      • Boundary fragments … working on focus.
      • Get contributions of 2D features externally
      • MSER … (Gary has code)
      • HoG
      • Star is already in and in for ROS build
    • Alpha channel, alternatives to contours RLE
      • New alpha transparency (per pixel) added
      • Introduce functions with advance ways of blending
      • Mask channels
      • Optical flow
    • 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation
      • SPIN
      • Radu is organizing his code into a library … maybe we can just lift this
    • 3D capture http://www.3dsom.com/index.html
    • VSLAM
      • Wait on what Kurt and James get together
    • Image stitching
    • Optical flow
      • Move code derived form former work in OpenCV
    • MLL
      • Extreme trees are coming
      • Reduce memory footprint
      • Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
      • Regression tests … maybe general cross validation
    • Misc: Focus detector, adaptive skin tracking
      • As discussed above
  3. Optimizations
    • .
  4. Documentation
    • .
  5. High GUI
    • .

OTHER

Action Items

NEW

Gary

  • Write back up note for 3D capture software

Vadim

  • Send small vector results for kd-tree
  • Stereo block matching improvements
  • Assess opencv use with numpy

Victor

FROM LAST TIME

Gary

Vadim

  • Check kdd trees on small vectors

Victor

  • Send link for application section

2009-04-14

Agenda

OpenCV Summer Release

  • Feature freeze end of May, June for test and doc.
  • General Structure:
    • Decisions on library structure (Google’s request, separate packages as is …)
      • Provide way to build OpenCV statically, then size varies with binary size
  • New Features
    • C++ interface
      • cxcore
      • cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
    • Better python
      • Vadim, Alex
    • 2D features C-descriptor, HoG Boundary Fragments …
      • C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
        • No re-implementation yet. Code based on randomized tree, but specialized version.
      • Boundary fragments … working on focus.
      • Get contributions of 2D features externally
      • MSER … (Gary has code)
      • HoG
      • Star is already in and in for ROS build
    • Alpha channel, alternatives to contours RLE
      • New alpha transparency (per pixel) added
      • Introduce functions with advance ways of blending
      • Mask channels
      • Optical flow
    • 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation
      • SPIN
      • Radu is organizing his code into a library … maybe we can just lift this
    • 3D capture http://www.3dsom.com/index.html
    • VSLAM
      • Wait on what Kurt and James get together
    • Image stitching
    • Optical flow
      • Move code derived form former work in OpenCV
    • MLL
      • Extreme trees are coming
      • Reduce memory footprint
      • Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
      • Regression tests … maybe general cross validation
    • Misc: Focus detector, adaptive skin tracking
      • As discussed above
  • Optimizations
    • .
  • Documentation
    • .
  • High GUI
    • .

Minutes

Vadim

  1. Converted image I/O to the new API, added extra parameters to cvSaveImage and its new counterpart that can regulate jpeg image quality, PNG compression.
    Added in-memory image I/O: cvDecodeImage & cvEncodeImage. For jpeg’s and PNG’s a special implementation has been done. For other formats a temporary file is used.
    Later on, the temporary files can be eliminated.
  2. Played a bit with block correlation stereo correspondence.
    Currently, the following scheme is used:
    1. first, the disparity map is computed with large block size = bigWSZ (e.g. 21×21).
    2. then, the disparity map is computed with smaller block size smallWSZ (like 11×11) and if at some pixel the new correspondence value C is within certain [A, B] range, where A is minimum and B is maximum, computed over bigWSZ x bigWSZ neighborhood.
      the results on a couple of stereo pairs are attached.
  3. Continued the work on porting optical flow to the new API and improving it. Unfortunately, the newest BBW code seem to be left at Intel, need to reproduce it from scratch. For now concentrated on improving the sparse optical flow.
  4. Purchased small lighting system (cube + 3 lamps) for the further experiments with 3D capturing.
  5. Anatoly is on vacation this week.

Victor

  • Another version of chamfer matching with templates represented as a binary image (as opposed to vector CvSeq representation) has been implemented. (Victor)
  • MLL: data loader has been implemented. Now one can load a dataset, learn a classifier and calculate test error with 3 lines of code!
  • CvMLData, new class. And in sample session TreeEngine.cpp for examples of loading and training.
    • Split into train and test, feature selection, missing values … all inside this class
    • Works for any tree based classifier (ExtremelyRandom forest)
  • Will update latest_tested_snapshot flag today.
  • New in memory I/O (with jpg or png) is implemented
    • Returns byte vector, template vector class (opencv version — has reference counting etc).
  • Support James Bowman on python interface
  • Benchmarks for kd-trees.
    • 3D space, FLANN is much faster
    • When dimensionality grows, difference is zero or OpenCV trees become faster
      • Might be memory allocation overhead
  • 2 possible Experts for stereo
  • Bought lighting for 3D, segmentation works well
    • Need the 3D license

Action Items

NEW

Gary

  • Write about 3D license.
  • Grep through ROS/vision for usage of OpenCV

Vadim

  • Send the experimental stereo code to Kurt
  • Send Gary email when the latest_tested_snapshot flag is moved
  • Need to check kd tree benchmarks
  • Vadim communicate May feature freeze for OpenCV and June for test and doc

Victor

  • Send link to application section
  • We need to update:
  • Accomplishments
  • Monthly announce
  • Change Logs Can we get new functions listed here?
  • Contributors credit page?
  • OpenCV_Team: Update what people are working on.

Kurt

  • Send better explanation for small-large/inner-outer windows

FROM LAST TIME

Gary

  • Write back up note for 3D capture software

Vadim

  • Send small vector results for kd-tree
  • Stereo block matching improvements
  • Assess opencv use with numpy

Victor

  • Link to applications’ section

2009-04-21

Agenda

  • Improving stereo progress report
  • Possible to get Navneet Dalal’s original HOG code http://pascal.inrialpes.fr/soft/olt/
  • New API update and time-lines
  • Python interface with James Bowman
  • Remember to touch and check-in the OpenCV ROS Makefile when you update latest_tested_snapshot SVN flag.
  • Machine learning
    • Nister trees
    • Plans for a robot vision architecture
      • Can we minimize up the memory footprint in nearest neighbor type apps?
  • Speed up to Chamfer matching
  • (Gary) Exploring new 3D feature that approximates surface normals
  • Fast segmentation

OpenCV Summer Release

  • Feature freeze end of May, June for test and doc.
  • General Structure:
    • Decisions on library structure (Google’s request, separate packages as is …)
      • Provide way to build OpenCV statically, then size varies with binary size
  • New Features
    • C++ interface
      • cxcore
      • cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
    • Better python In progress thanks to James Bowman
      • Vadim, Alex
    • 2D features C-descriptor,
      • HoG ‘’’(status)‘’’
        • Boundary Fragments … In progress thanks to Marius Muja
      • C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
        • No re-implementation yet. Code based on randomized tree, but specialized version.
      • Get contributions of 2D features externally
      • MSER … (Gary has code)
      • HoG
      • Star is already in and in for ROS build
    • Alpha channel, alternatives to contours RLE
      • New alpha transparency (per pixel) added
      • Introduce functions with advance ways of blending
      • Mask channels
      • Optical flow
    • 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation, normals
      • SPIN
      • Radu is organizing his code into a library … maybe we can just lift this
    • 3D capture http://www.3dsom.com/index.html
    • VSLAM
      • Wait on what Kurt and James get together
    • Image stitching
    • Optical flow
      • Move code derived form former work in OpenCV
    • MLL
      • Extreme trees are coming
      • Reduce memory footprint
      • More convenient training via a data class
      • Need to work on input and output of data in CSV (or other simple deliminators), XML and YML
      • Regression tests … maybe general cross validation
    • Misc: Focus detector, adaptive skin tracking
      • As discussed above
  • Optimizations
    • .
  • Documentation
    • .
  • High GUI
    • .

Minutes

Vadim

  1. Image encoding/decoding has been further improved: for bmp, ppm and tiff the codecs do not use temporary files. Performance of PNG encoder has been improved by factor of 5 by tuning the compression parameters passed to libpng.
  2. latest_tested_snapshot tag has been finally updated. Prior to this, OpenCV has been built and tested on MacOSX, 64-bit Linux and Windows. The update includes all the recently added functionality, including slightly improved block matching-based stereo correspondence.
  3. added new C++ classes for multi-dimensional matrixes, cv::MatND and cv::SparseMat and the derived template classes with convenient operators for element access. The classes will be used to port histogram processing to C++. For cv::MatND the set of basic operations (arithmetics, statistics etc.) has been added too, 45 functions in total. The functions just process the n-D matrices plane by plane using the corresponding functions operating on cv::Mat’s.
  4. About 70% done with new API
    • Histograms, contours, polar, obj detection maybe (probably a thin wrapper since extending the object detectors)
  5. Conversion of MLL would be next
  6. High GUI should be easy to convert
  7. Anatoly is back from vacation, we started design and implementation of 3D reconstruction code. For feature matching we are going to use SURF with outlier filtering based on epipolar geometry constraits. For the optimization part the existing Levenberg-Marquardt solver from OpenCV.
  8. Continued work on improving sparse optical flow algorithm and porting it to C++ interface. The draft version is ready. Testing is to be done this week.
  9. Stereo sample code is with Kurt, haven’t done anything until hear back.
    • Small windows did improve a bit

Coming:

  • Improved Haar training
  • 3D reconstruction

Victor

  • Maria looking at Adaboost cascade detector trying to make it work with arbitrary features instead of just Haar wavelets.
    • This code is separate from MLL classifiers and specialized to cascades so we can’t just move general boosting in.

Python

  • wrapper code for py→c can maybe extend for py→C++ for James python implementation

MLL for robots. Want:

  • Combine separate learning in way that grows sublinearly in recognition time.
    • Willing to use more memory for this
  • For features. Needs to deal with
    • Missing features
    • Newly added features
  • Want some form of memory optimization that can be done offline.

Action Items

NEW

Gary

  • Adaboost with cascade comparison? Papers?
  • Send HOG code to Vadim
  • 3D capture license …

Vadim

  • Look at moving Navneet Dalal’s original HOG code http://pascal.inrialpes.fr/soft/olt/ into OpenCV replacing its data structures and learning with OpenCV and MLL structures and functions. Maria?

Victor

  • We need to update:
    • Accomplishments
    • Monthly announce
    • Change Logs Can we get new functions listed here?
    • Contributors credit page?
    • OpenCV_Team: Update what people are working on.

FROM LAST TIME

Gary

  • Write about 3D license.
  • Grep through ROS/vision for usage of OpenCV

Vadim

  • Send the experimental stereo code to Kurt
  • Send Gary email when the latest_tested_snapshot flag is moved
  • Need to check kd tree benchmarks
  • Vadim communicate May feature freeze for OpenCV and June for test and doc

Victor

  • Send link to application section

2009-04-27

Agenda

  • New API progress
  • Python progress
  • Optical Flow progress
  • Fast Graph-Cut based Segmentation — Konstantin Mirin might do this, we’ll see.
    • Based on A Multilevel Banded Graph Cuts Method for Fast Image Segmentation
      • step.polymtl.ca/~rv101/bandedgraphcuts.pdf
  • MLL
  • Utility function (?) Specify translation and angle change from face on normal and return equivalent homography matrix.
  • Maruis is fairly close to being able to contribute fast Chamfer matching with orientation.
    • Another function is simply robust orientation from noisy line segments.

OpenCV Summer Release

  • Feature freeze end of May, June for test and doc.
  • General Structure:
    • Decisions on library structure (Google’s request, separate packages as is …)
      • Provide way to build OpenCV statically, then size varies with binary size
  • New Features
    • C++ interface
      • cxcore
      • cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
    • Better python In progress thanks to James Bowman
      • Vadim, Alex
    • 2D features C-descriptor,
      • HoG ‘’’(status)‘’’
        • Boundary Fragments … In progress thanks to Marius Muja
      • C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
        • No re-implementation yet. Code based on randomized tree, but specialized version.
      • Get contributions of 2D features externally
      • MSER … (Gary has code)
      • HoG
      • Star is already in and in for ROS build
    • Alpha channel, alternatives to contours RLE
      • New alpha transparency (per pixel) added
      • Introduce functions with advance ways of blending
      • Mask channels
      • Optical flow
    • 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation, normals
      • SPIN
      • Radu is organizing his code into a library … maybe we can just lift this
    • 3D capture http://www.3dsom.com/index.html
    • VSLAM
      • Wait on what Kurt and James get together
    • Image stitching
    • Optical flow
      • Move code derived form former work in OpenCV
      • Variational optical flow (maybe)
    • MLL
      • Extreme trees are coming
      • Reduce memory footprint
      • More convenient training via a data class
      • Regression tests … maybe general cross validation
    • Misc: Focus detector, adaptive skin tracking
      • As discussed above
  • Optimizations
    • .
  • Documentation
    • .
  • High GUI
    • In memory .jpg, .png

Minutes

Vadim:

  1. Added new-style histogram functions. Besides the use of the new C++ multidimensional matrix classes, the functions can now compute histograms or back projections directly from color images, there is no need to split an image into separate color planes.
  2. Finished improved sparse optical flow. The new function can use color information and spatial image derivatives for better accuracy. Rewritten cvCalcOpticalFlowBM after James’ bug report. The new implementation can still be improved a lot, e.g. some modern algorithm used in video codecs can be implemented.
  3. Fixed several bugs in the new code: bug in Farnerback’s optical flow with non-squared images (patch by Ballard Stonewall), bug in reshape (by asmwarrior), bugs in cvCalcOpticalFlowBM and cvGetAffineTransform (by James Bowman),
    bug in morphology (by Marius). Another bug in highgui about capturing rectified stereo from Videre camera has been discovered but not fixed yet.
  4. Anatoly continued the work on feature registration part for 3D reconstruction project. We used the recently purchased lighting setup to capture some objects with nicely separated background, and find the corresponding feature points. We tried SURF with different parameters and cvGoodFeaturesToTrack + cvCalcAffineFlowPyrLK. When for every pair of views for which we found many enough corresponding pairs of features we estimated fundamental matrix and used it to filter outliers. Unfortunately, we found that normally only about 10 pairs satisfy the epipolar constraints and that’s for a good reason – SURF often failed to find the correct corresponding points; affine flow’s “hit ratio” was higher, but the number of features was less than in case of SURF, so the total number of good matches was about the same. May be we should take SURF keypoints and try affine flow on it, or try another descriptor, or preprocess images somehow to remove noisy features. The experiments continue.

Plans for this week:

  1. Finish C++ interface for cv module (contours, color conversion; object probably, camera calibration and object detection too)
  2. Update latest_tested_snapshot tag
  3. Study the HOG people detector
  4. Advance in feature matching for 3D reconstruction.

API

  • Contours
  • About a dozen more, so 1 week for cv to be finished
    • Then to decide on MLL change over.
      • Convert decision tree methods, K nearest too

HOG

Cascade generalization

  • Maria custom class for data retrieval
  • Idea was to extract previous re-write of the tree code
    • Now its done via call-backs
      • Allows running in or out of memory
      • Issue of how to do from a user stand-point. Positive samples are obvious, negatives are harder since we have to provide filtered (surviving) negative samples to later stages.
  • For now, want to just rebuild haar training with this. 2 weeks: Implement and debug. Then 1 week/feature.
  • Training data
    • INRIA dataset on line
      • Convert

Optical flow

  • BM fixed and done
  • Sparse optical flow can work with color now
  • Lost variational optical flow (Horn Schunk with local property)

Action Items

NEW

Gary

  • Talk to Kurt about expanding the team for stereo
  • Ask James about python progress
  • 3D capture license …

Vadim

  • HOG image code
    • Look at moving Navneet Dalal’s original HOG code http://pascal.inrialpes.fr/soft/olt/ into OpenCV replacing its data structures and learning with OpenCV and MLL structures and functions. Maria?

Victor

All:

  • Read the ICCV 2009 paper
  • We need to update:
    • Accomplishments
    • Monthly announce
    • Change Logs Can we get new functions listed here?
    • Contributors credit page?
    • OpenCV_Team: Update what people are working on.

FROM LAST TIME

Gary

  • Adaboost with cascade comparison? Papers?
  • Send HOG code to Vadim
  • 3D capture license …

Vadim

Victor

2009-05-05

Agenda

  • HOG code?
  • OpenCV Python. Now in ROS, how about into OpenCV?
  • Morphology bugs (James Bowmen reported)?
  • Is there news on dense optical flow? Get Michael Black’s code?
  • C++ conversion of cv?
    • contours, color conversion; object probably, camera calibration and object detection
  • 3D feature matching/3D reconstruction progress?
  • Parts based model from outlets into OpenCV?
  • Cascade generalization progress
  • 3D features for recognition (Victor)?
  • Normals etc for 3D point clouds in grids (as derived from dense stereo) (Gary)
  • Fast Chamfer matching with orientation into OpenCV (Marius)?

OpenCV Summer Release

  • Feature freeze end of May, June for test and doc.
  • General Structure:
    • Decisions on library structure (Google’s request, separate packages as is …)
      • Provide way to build OpenCV statically, then size varies with binary size
  • New Features
    • C++ interface
      • cxcore
      • cv — start with filtering functions, resize, optical flow, object detection (2 weeks)
    • Better python In progress thanks to James Bowman
      • Vadim, Alex
    • 2D features C-descriptor,
      • HoG ‘’’(status)‘’’
        • Boundary Fragments … In progress thanks to Marius Muja
      • C-descriptor … resolved issues with Michael, Vadim understands code more deeply (next week)
        • No re-implementation yet. Code based on randomized tree, but specialized version.
      • Get contributions of 2D features externally
      • MSER … (Gary has code)
      • HoG
      • Star is already in and in for ROS build
    • Alpha channel, alternatives to contours RLE
      • New alpha transparency (per pixel) added
      • Introduce functions with advance ways of blending
      • Mask channels
      • Optical flow
    • 3D features (Plane fit, PFH, FPFH from Radu), 3D Delaunay, 3D convex hull, mesh generation, normals
      • SPIN
      • Radu is organizing his code into a library … maybe we can just lift this
    • 3D capture http://www.3dsom.com/index.html
    • VSLAM
      • Wait on what Kurt and James get together
    • Image stitching
    • Optical flow
      • Move code derived form former work in OpenCV
      • Variational optical flow (maybe)
    • MLL
      • Extreme trees are coming
      • Reduce memory footprint
      • More convenient training via a data class
      • Regression tests … maybe general cross validation
    • Misc: Focus detector, adaptive skin tracking
      • As discussed above
  • Optimizations
    • .
  • Documentation
    • .
  • High GUI
    • In memory .jpg, .png

Minutes

Gary

  • Working on 3D features
    
    int computeNormals(const CvMat *xyzd, const CvPoint &p, vector<triangle_offsets> &v, float sqr_root_max_radius, float max_depth_difference,
    vector<float> &normal_pt3d, vector<vector<float> > &normals, vector<vector<float> > &normal_angles);

vector findMedianND(vector<vector > &v, int &median_index);

vector centerOfMassND(vector<vector > &v, vector *b = NULL);

Vadim

  1. fixed a few bugs in the new histogram functions; still camshiftdemo and chessboarddetector fail despite that histogram tests pass; so need to explore it.
  2. started porting contours and comp. geomery functions to the C++ API. In the new API a contour is represented as Vector or Vector, so it’s much easier to operate with.
  3. studied HOG training and detection code by Navneet Dalal. Here are my findings:
    1. There are precompiled binaries are available for Mac and Linux; the source code is also avaialble and it builds fine (on MacOSX at least).
      All the training data for people detection is also available. The training procedure runs fine on this training data.
      The code depends on Boost, Blitz++ and Imlib2 and includes a small subset of Vigra library.
      Also, the training pipeline uses modified SVMLight (the modification was done to make SVMLight work with huge amount of data,
      e.g. authors mention #7Gb buffers necessary for training). SVM training is run as a separate executable from within the shell script.
    2. The training is done as following:
      • HOG descriptors are computed for every positive sample and randomly selected windows from negative images. The default window size for person detection is 64×128. This is done by dump_rhog utility.
      • Then the dump file is converted via dump4svmlearn to the format acceptable by svm_learn from SVMLight
      • After this initial model is trained, it is passed to another utility “classify_rhog” that is run in a special mode. Normally, the utility is used to just use the trained model to do batch detection and optional visualization of the results. But in this mode it runs exhaustive search for “hard” negative samples, i.e. the false positives that the initial SVM model gave.
      • After that all the initially used positive and negative samples, and also as many “hard” negative samples as fit into the memory are passed (via dump_rhog and dump4svmlearn) o svm_learn that produces the final model.
      • Linear SVM is used by default and seems to be the only fully supported SVM type supported by the detection part. Authors say, they tried RDF kernel as well, but the increased training and detection time did not worth the small decrease of false positive scores. Thus, the result of all the training procedure is a single floating point vector M (With the default parameters it has 3781 elements). And the basic “binary” detection rule, once HOG descriptor d is computed, is “M^t*[d, 1] > c ? true : false”, where c is some parameter, that we set depending on the hit-rate/false positive-rate balance that we want to achieve.
      • On top of the binary detector, there is mean-shift filtering procedure that is used to group the found regions (and may be throw away some, depending on the computed confidence measures)
      • HOG computing in the PhD thesis and in the code is quite different from the code contributed to OpenCV:
        • the original HOG supports color images, OpenCV’s version does not
        • the original code can do the optional gamma normalization (turned on by default)
        • the original code applies Gaussian weighting function to the gradient values
        • the original code uses tri-linear interpolation when updating histograms to avoid/reduce aliasing effect.
        • the original code uses a bit more tricky histogram normalization (so called L2-hysteresis normalization, borrowed from D. Lowe paper on SIFT)
      • The whole code is quite difficult to read because of using multiple abstraction layers and many advanced C++ concepts. I suggest the following:
        • the training code is quite sophisticated, has many options and pretty stable. It can take quite a bit of time to reimplement it (although, if we only implement a subset of possible options, e.g. only the default mode, that may be faster to implement). But as the training is not an online procedure demanding realtime performance, we can reuse the existing training code as is, at least for now.
        • we can implement conversion utility from svm_learn output to an XML file in OpenCV-compatible format, that will contain both the decision boundary vector (M) and the HOG parameters.
        • the HOG descriptor in OpenCV can be extended to the original variant in a week (the default mode only for now). It will certainly be much, much faster than the original. That is, in a week we can have some fast people detector that we can evaluate and decide whether we should put extra efforts to porting the training application.

Plans for next week:

  • fix remaining bugs in histograms
  • finally update latest_tested_snapshot
  • extend HOG descriptor in OpenCV and try to make a sample for the people detector
  • stereo experiments

HOG

  • Triggs code modifies SVM light to work with large data ~ pseudo perceptron mode
  • Detection is dot-product
  • Can make detection 10x or more faster
    • Keep original training for now, don’t understand
  • Speed up detector and allow specific window location search mode

Python for OpenCV

  • Separate sourceforge project
    • 30 tests
  • would like more tests
    • If finds bug, but them in test
  • Wiki isn’t up to date
  • People use ctypes
    • Point to James on wiki
  • Right now, we have the swig interface

Documentation

  • html is most up to date
  • James to look into

Morphology bug

  • fixed the image shift bug
  • Will update latest tested shapshot

Dense flow

  • Tomnorrow will try to get dense optical flow

C++

  • This week will be finished

3D Reconstruction

  • Resume on May 11 when Anatoly gets back

Cascade generalization (Maria)

  • Rewriting haar training application
  • Extend decision tree
    • Support data that will not fit in memory using callbacks
  • Finish this week … will send estimate on finishing cascade

Outlets

  • One way descriptor Fua et al
    • Include into OpenCV
  • Perona, constellation model

Action Items

NEW

Gary

  • Look at license for HOG and see about use of code
  • Pointer to new python interface

Vadim

  • Look at license
  • Send update esitmate for finishing cascade

Victor

James

  • Look into cocumentation solutions

All:

  • Read the ICCV 2009 paper
  • We need to update:
    • Accomplishments
    • Monthly announce
    • Change Logs Can we get new functions listed here?
    • Contributors credit page?
    • OpenCV_Team: Update what people are working on.

FROM LAST TIME

Gary

  • Talk to Kurt about expanding the team for stereo (not done)
  • Ask James about python progress.
  • 3D capture license … (darn)

Vadim

  • HOG image code
    • Look at moving Navneet Dalal’s original HOG code http://pascal.inrialpes.fr/soft/olt/ into OpenCV replacing its data structures and learning with OpenCV and MLL structures and functions. Maria?

Victor

  • One way descriptor into OpenCV
    • Haven’t

All:

  • Read the ICCV 2009 paper
  • We need to update:
    • Accomplishments
    • Monthly announce
    • Change Logs Can we get new functions listed here?
    • Contributors credit page?
    • OpenCV_Team: Update what people are working on.

2009-05-26

Agenda

  • Update HOG implementation
    • Flexible Cascade
  • Update Python
    • What’s the coverage? doc plans? examples?
  • Code contribution — official plan
    • Test, docs etc
  • Update Doc
  • Summer Release
    • Code freeze
    • Need to catch up test/regression code
    • Need to switch to and revamp the documentation

OpenCV Summer Release

  • Feature freeze end of May, June for test and doc.
  • General Structure:
    • :\ ? Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
  • New Features
    • C++ interface
      • {OK} cxcore
      • cv — start with filtering functions, resize, optical flow, object detection
    • Better python interface
      • ‘’In progress thanks to’’ (’’’*James Bowman*’’’)
    • 2D features
      • {OK} HOG
        • Boundary Fragments … In progress thanks to Marius Muja
      • C-descriptor … (Vadim)
        • :\ ? See below for LePetit style randomized tree.
      • {OK} MSER
      • {OK} Star
    • {X} Alpha channel, alternatives to contours RLE
      • {X} New alpha transparency (per pixel) added
      • {X} Introduce functions with advance ways of blending
      • {X} Mask channels
    • 3D features
      • Plane fit (Gary from Radu’s code)
      • LePetit style one way feature (Victor)
      • {X} PFH, FPFH from Radu,
      • {X} 3D Delaunay,
      • {X} 3D convex hull,
      • {X} mesh generation,
      • normals in grid (Gary)
      • normal based 3D patch (Gary)
      • {OK} SPIN
      • {X} 3D capture http://www.3dsom.com/index.html
      • {X} VSLAM
      • {X} Image stitching
    • Optical flow
      • {OK} Dense (Farnerback)
      • {X} Variational optical flow
    • MLL
      • {OK} Extreme trees
      • {OK} Reduce memory footprint
      • {OK} More convenient training via a data class
      • Nister trees (Patrick)
      • FLANN or appropriate parts (Marius)
      • :\ ? LePetit style random trees
  • Misc:
    • {X} Focus detector,
    • {OK} adaptive skin tracking
  • Optimizations
    • Various functions, ongoing …
  • Documentation
    • Latex
  • High GUI
    • In memory .jpg, .png
    • {X} move to C++ interface
  • Testing functions
    • Regression tests for all functions
  • Define user contribution process:

Minutes

Vadim Update:

  1. HOG has been opimized by factor of ~4. Instead of #8sec it now processes person_191 image from INRIA database in less than 0.5sec. The HOG has been integrated into the library, demo sample opencv/samples/c/peopledetect.cpp has been added.
  2. The taged branch, latest_tested_snapshot, has been updated to include various bug fixes and the HOG detector.
  3. 9 bugs have been fixed: 2 bugs in drawing functions (reported by James and others), 2 bugs in cvFindFundamentalMat (one of them was reported by Anatoly), bug in cvSolve (unnecessary assert()), bug in cvConvertMaps (reported by Caroline), 3 bugs in build scripts. All the fixes are put into the tested branch as well.
  4. Maria finished rewriting the haartraining application and now debugging it.
  5. Anatoly continues work on the image registration part for 3D reconstruction. Several problems (including bugs in cvFindFundamentalMat) have been fixed, but the results are not satisfactory yet to move further.

HOG

  • In SVN and latest_stable
  • Needs commit access to ROS package
  • Training is original still
  • 2.5×4 = 10 faster (took 4 seconds, latest version takes 0.5)

Cascade

  • Maria finished more flexible but
    • right now Haar only right now
  • Debugging
    • override features method
    • General boosting code will end up somewhat different
    • Darrius (wife Alexander Sorokin) balancing between tree depth and gradient boosting
      • Open code, will check out

Python

  • ~300 functions
  • No machine learning
  • No cvaux
    • Skipped tricky functions
    • Many bugs
  • Samples are being done
  • Return’s output parameters
    • new is much more “pythonic”
  • Python bindings (gen.py)
    • Not using ctypes
    • Can be adapted to new C++?
      • Can make that switch (in James’ spare time)

Documentation

  • C, C++, Python switchable.
  • Just place holders
  • Standardize to LaTex
  • Wiki pointer for documentation bugs

cvaux

  • SPIN
  • Background
  • and older stuff like HMM faces

Code coverage

  • gcov for coverage
    • bullseye was used at Intel
  • MLL covered for decision trees
  • New functionality SPIN, HOG, Stereo NOT covered
  • Covered is simple image processing, Matrix functions
  • Complex functions — how to implement a test

Summer

  • OUT:
    • Image stitching
    • SLAM
    • Bundle adjustment
      • Estimation of 3D coords
      • Bundle adjustment
    • Alpha blending
    • Focus detector
    • UI enhancements in HighGUI
    • PFH will wait for Radu to be here
  • IN
    • 3D Capture
      • Image registration, 2D projection to get tracks for SFM
      • MV Geometry pipeline
        • Estimation of 3D coords
    • Randomized trees
      • Random forest
      • Extremely Randomized Trees
      • c-descriptors, one way trees
    • Gunnar Farneback dense flow is in
    • Grid based normals
      • Feature
    • Skin detector
    • SPIN
    • Codebook
    • HighGUI
      • In memory jpg
      • Stereo camera support (with optional rectification, check with point gray)
    • Nister tree across
    • *Marius*’s library

Action Items

Vadim

  • Figure out code contribution process
  • Put up link for bugs and documentation bugs

Gary

  • 3D features
  • Get Vadim write access to ROS packages
  • Ask Alexander about Darrius’s gradient boosting code
  • Figure out code contribution process (namespace, datastructures (use OpenCV)
  • Put up link for bugs and documentation bugs

Victor

  • One way descriptor

Marius

  • Fast Chamfer with orientation

Patrick

  • Nister trees in

All:

  • We need to update:
    • Accomplishments
    • Monthly announce
    • Change Logs Can we get new functions listed here?
    • Contributors credit page?
    • OpenCV_Team: Update what people are working on.

2009-06-02

Agenda

  • Orientation filters for features from SIFT (“featureOrientation2D()”)
  • updates
    • 3D feature
    • boundary
    • Vadim
    • Docs
  • Code freeze and general status
  • Code contribution

OpenCV Summer Release

  • Feature freeze end of May, June for test and doc.
  • General Structure:
    • :\ ? Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
  • New Features
    • C++ interface
      • {OK} cxcore
      • cv — start with filtering functions, resize, optical flow, object detection
  • Better python interface
    • ‘’In progress thanks to’’ (’’’*James Bowman*’’’)
  • 2D features
    • {OK} HOG
      • Boundary Fragments … In progress thanks to Marius Muja
    • C-descriptor … (Vadim)
      • :\ ? See below for LePetit style randomized tree.
    • {OK} MSER
    • {OK} Star
    • Self-similar detector
  • {X} Alpha channel, alternatives to contours RLE
    • {X} New alpha transparency (per pixel) added
    • {X} Introduce functions with advance ways of blending
    • {X} Mask channels
  • 3D features
    • Plane fit (Gary from Radu’s code)
    • LePetit style one way feature (Victor) (to Anatoly)
    • {X} PFH, FPFH from Radu,
    • {X} 3D Delaunay,
    • {X} 3D convex hull,
    • {X} mesh generation,
    • normals in grid (Gary)
    • normal based 3D patch (Gary)
    • {OK} SPIN
    • {X} 3D capture http://www.3dsom.com/index.html
    • {X} VSLAM
    • {X} Image stitching
  • Optical flow
    • {OK} Dense (Farnerback)
    • {X} Variational optical flow
  • MLL
    • {OK} Extreme trees
    • {OK} Reduce memory footprint
    • {OK} More convenient training via a data class
    • Nister trees (Patrick)
    • FLANN or appropriate parts (Marius)
    • :\ ? LePetit style random trees
    • Generalized (boosted) cascade (generalizing Haar cascade)
  • Misc:
    • {X} Focus detector,
    • {OK} adaptive skin tracking
  • Optimizations
    • Various functions, ongoing …
  • Documentation
    • Latex
  • High GUI
    • In memory .jpg, .png
    • {X} move to C++ interface
  • Testing functions
    • Regression tests for all functions
  • Define user contribution process:

Minutes

  • Put in feature orientation
    • Use SURF or SIFT like technique as a general convenience to find gradient orientations at points in an image
  • Rainer has contributed an implementation of self-similar features for OpenCV

Vadim

  1. Resumed the work on integrating C-descriptor into OpenCV. Specifications have been prepared, ~30% of the conversion work has been done (data structures design, outline of the internal functions etc.)
  2. Anatoly continues working on the image registration part of 3D rec. package. The picture with the latest results is attached. Here we use one-way descriptor, received from Victor. The results are notably better than when using SURF descriptor, but there are not many matches. Hopefully, a better keypoint detector could help here. {{"correspondence.png}}":images/correspondence.png}}
  3. Maria pretty much finished debugging haartraining application. Now it’s quite stable and achieves the target hit rate and false alarm rate on our training dataset. Maria is now adapting the face detection example to use the newly trained cascade (as the data structure is different).

Rainer Lienhart

self-similarity feature for the summer release of opencv. The documentation should mention the following two papers:

  1. Shechtman, E., Irani, M.: Matching local self-similarities across images and videos. CVPR, (2007)
  2. Eva Horster, Thomas Greif, Rainer Lienhart, Malcolm Slaney. Comparing Local Feature Descriptors in pLSA-Based Image Models. 30th Annual Symposium of the German Association for Pattern Recognition (DAGM) 2008, Munich, Germany, June 2008.

I also put the PDFs of these two papers into the top-level directory for reference.

At the moment the self-similarity features are not rotation-invariant. This could be changed in future if needed.

Everything in “src/” should be compiled and linked. It only needs opencv as an external dependency. I used Eclipse to compile everything, so you might want to use the project file.

I uncommented all IPP call and replaced them with slow c-loops. If you you have IPP it is easy to change back.

I also put some documentation of the various data types into the main directory. It’s called MMC Library Reference Manual.

Action Items

Gary

  • Find tools for code contribution

LAST TIME

Vadim

  • Figure out code contribution process
  • Put up link for bugs and documentation bugs

Gary

  • 3D features
  • Get Vadim write access to ROS packages
  • Ask Alexander about Darrius’s gradient boosting code
  • Figure out code contribution process (namespace, datastructures (use OpenCV)
  • Put up link for bugs and documentation bugs

Victor

  • One way descriptor

Marius

  • Fast Chamfer with orientation

Patrick

  • Nister trees in

All:

  • We need to update:
    • Accomplishments
    • Monthly announce
    • Change Logs Can we get new functions listed here?
    • Contributors credit page?
    • OpenCV_Team: Update what people are working on.

2009-06-09

Agenda

  • Phase space contributing Sergey’s Bundle adjustment
  • Self similarity feature
  • James’s document conversion to Latex→Sphinx→html
  • Unit testing code (gtest?)
  • 3D registration, Haar training
  • Documentation
    • http://opencv.willowgarage.com/documentation/index.html
  • Python
  • HOG/Cascade

OpenCV Summer Release

  • Feature freeze end of May, June for test and doc.
  • General Structure:
    • :\ ? Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
  • New Features
    • C++ interface
      • {OK} cxcore
      • cv — start with filtering functions, resize, optical flow, object detection
  • Better python interface
    • In progress thanks to James Bowman
  • 2D features
    • {OK} HOG
      • Boundary Fragments … In progress thanks to Marius Muja
    • C-descriptor … (Vadim)
      • :\ ? See below for LePetit style randomized tree.
    • {OK} MSER
    • {OK} Star
    • Self-similar detector
  • {X} Alpha channel, alternatives to contours RLE
    • {X} New alpha transparency (per pixel) added
    • {X} Introduce functions with advance ways of blending
    • {X} Mask channels
  • 3D features
    • Bundle Adjustment (probable contribution by Phase Space)
    • Plane fit (Gary from Radu’s code)
    • LePetit style one way feature (Victor) (to Anatoly)
    • {X} PFH, FPFH from Radu,
    • {X} 3D Delaunay,
    • {X} 3D convex hull,
    • {X} mesh generation,
    • normals in grid (Gary)
    • normal based 3D patch (Gary)
    • {OK} SPIN
    • {X} 3D capture http://www.3dsom.com/index.html
    • {X} VSLAM
    • {X} Image stitching
  • Optical flow
    • {OK} Dense (Farnerback)
    • {X} Variational optical flow
  • MLL
    • {OK} Extreme trees
    • {OK} Reduce memory footprint
    • {OK} More convenient training via a data class
    • Nister trees (Patrick)
    • FLANN or appropriate parts (Marius)
    • :\ ? LePetit style random trees
    • Generalized (boosted) cascade (generalizing Haar cascade)
  • Misc:
    • {X} Focus detector,
    • {OK} adaptive skin tracking
  • Optimizations
    • Various functions, ongoing …
  • Documentation
    • Latex
  • High GUI
    • In memory .jpg, .png
    • {X} move to C++ interface
  • Testing functions
    • Regression tests for all functions
  • Define user contribution process:

Minutes

  • Phase Space, via Sergey and Xavier DeLacour
    • Are probably going to contribute bundle adjustment to OpenCV
    • Xavier also contributed Octave wrappers, KD Trees
  • Stereo demo. Updated.
  • New interface almost finished for C++ interface
  • Self similarity … not looked at yet
    • Not rotation invarient
  • For 2D rotation invariance
    • Transform each patch using random rotations
    • Compute matches for each
    • Choose best
  • Documentation http://opencv.willowgarage.com/documentation/index.html
    • Located in doc2’
    • move to doc
    • Need to update the Wiki
    • Can turn on document build in cmake configure stage
  • Unit testing
    • Current OpenCV system cxts module
      • Cxcore (lots) 50-75%
      • Cv (lots) 30%
      • Mll (little) ?
      • C++ interface (all of Cxcore, half of Cv) 3/4 coverage
    • Not covered
      • New functions
      • C++ interface
    • Python tests 300
      • Test files about 100
  • Python
    • Bugs that are bottleneck
  • HOG/People
    • Pass a mask to the detect function valid top left corners
  • New Cascade
    • Have been trying to reproduce old cascade using ~small database
    • Now are trying the whole face database
      • This week ~ might be done
    • Current haar training uses whole dataset, can we do online training

Vadim

  1. Nearly finished converting cv module to C++ interface. The converted functions are:
    Canny, cvtColor, motion templates, CAMShift + Meanshift, Kalman filter, floodfill, watershed, inpainting, distance transform, Hough transform (circle & line detection), moments, findContours, contour processing functions (like approxPoly, convexHull, contourArea etc.), Haar object detection, block matching stereo, feature detectors (Star, SURF, MSER, GoodFeaturesToTrack), 3D geometry (findHomography, findFundamentalMat, solvePnP …) and calibration functions.
    There remaning non-yet converted functions in cv are:
    Planar subdivisions (it’s unclear what to do with the data types)
    cvCalcEMD2, cvPyrSegmentation, cvPyrMeanShiftSegmentation, cvCalcOpticalFlow{LK|BM|HS} – old optical flow algorithms,
    ConDensation algorithm (probably, need to be moved to cvaux or extended), cvSnakeImage,
    kd-tree (need to revise the implementation), POSIT, graph cuts-based stereo (needs some optimization)
    + about a half-dozen of other small functions.
    Except for kd-tree, EMD and maybe couple of other functions, the rest will probably be converted after the release.
  2. Anatoly works on integration of the one-way descriptor to OpenCV. 1 serious bug has been found and fix is sent to Victor.
  3. Maria wrote a converter from the new cascade XML representation to the old one, supported by OpenCV functions. Several bugs have been fixed in haartraining that keeps getting better.

Plans:

  • finish integration of C-descriptor
  • convert kd-trees to the new interface or take it from FLANN.
  • finalize the new haartraining and replace the old version with the new one (there is a risk that we can not stabilize it enough for the release, probably, move it to the next after 2.0 update)
  • figure out if MLL classes can be quickly adjusted to be compatible/easier to use with the new C++ data structures.

Victor

  • On vacation

Action Items

Vadim

  • Look at priority 6 python bugs
    • Will look at today

James

  • Merge doc2 to doc
  • In generated html
    • Document bug button(s)
      • OR: Other ways of getting feedback such as a comment sections
    • link back to OpenCV Wiki
    • cmake to build

Caroline

  • Email mask interface considerations

Done

Not done:

  • 3D features
  • Nistor Tree
  • Figure out code contribution process
  • Documentation bug link
  • Fast Chamfer with orientation

2009-06-16

Agenda

  • Ongoing status
  • New docs up
  • Bundle adjustment from phase space
  • Contribution template discussion
  • 3D scan progress
  • Where is HOG training code
  • Some post release thoughts —
    • more concentation on 2D and 3D features, object recognition
    • Ability to run in cloud

OpenCV Summer Release

  • Feature freeze end of May, June for test and doc.
  • General Structure:
    • :\ ? Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
  • New Features
    • C++ interface
      • {OK} cxcore
      • cv — start with filtering functions, resize, optical flow, object detection
  • Better python interface
    • In progress thanks to James Bowman
  • 2D features
    • {OK} HOG
      • Boundary Fragments … In progress thanks to Marius Muja
    • C-descriptor … Vadim
      • :\ ? See below for LePetit style randomized tree.
    • {OK} MSER
    • {OK} Star
    • Self-similar detector
  • {X} Alpha channel, alternatives to contours RLE
    • {X} New alpha transparency (per pixel) added
    • {X} Introduce functions with advance ways of blending
    • {X} Mask channels
  • 3D features
    • Bundle Adjustment (probable contribution by Phase Space)
    • Plane fit (Gary from Radu’s code)
    • LePetit style one way feature (Victor) (to Anatoly)
    • {X} PFH, FPFH from Radu,
    • {X} 3D Delaunay,
    • {X} 3D convex hull,
    • {X} mesh generation,
    • normals in grid (Gary)
    • normal based 3D patch (Gary)
    • {OK} SPIN
    • {X} 3D capture http://www.3dsom.com/index.html
    • {X} VSLAM
    • {X} Image stitching
  • Optical flow
    • {OK} Dense (Farnerback)
    • {X} Variational optical flow
  • MLL
    • {OK} Extreme trees
    • {OK} Reduce memory footprint
    • {OK} More convenient training via a data class
    • Nister trees (Patrick)
    • FLANN or appropriate parts (Marius)
    • :\ ? LePetit style random trees
    • Generalized (boosted) cascade (generalizing Haar cascade)
  • Misc:
    • {X} Focus detector,
    • {OK} adaptive skin tracking
  • Optimizations
    • Various functions, ongoing …
  • Documentation
    • Latex
  • High GUI
    • In memory .jpg, .png
    • {X} move to C++ interface
  • Testing functions
    • Regression tests for all functions
  • Define user contribution process:

Minutes

  • Trying to push documentation bugs to bug reporting system
  • CvMat Mat Documentation
  • unit tests
    • Use old cxcoretest, cvtest,
    • Put new tests in python (?)
      • Contact James
  • Haar training seems to be what it was
    • Finalize to release
    • Add new features
      • Implement new class from base class
      • Compute feature on image (call back)
      • Implement read write functions: 2 phase: boosted emsembles, then feature descriptions
      • Enumerate the features. Current Haar sets rect size and then enumerates all possible features
        • Example should go into documention
    • but probably not prior to release
  • Need to slip code release due to major items 2 weeks
    • The remaining things to be done (all of the items below can be done in parallel, but I have to spend some time on each of the items):
      • kdTree cleanup (now it does not pass the tests) or possible FLANN integration
      • debugging C-descriptor (1-2 days)
      • integration of one-way descriptor. Integration of the fixes from Victor (~1 week)
      • code review of the rewritten haartraining, ingtegration (~1 week)
      • bundle adjustment integration (1 week at least, may be more)
      • Nister trees integration (1 week at least)
  • 3D scan code
    • Stuck at registration part: Need to collect many correspondences per view
      • Trying different descriptors — one way descriptor seems better than SIFT and SURF for this
      • Need good, dense keypoint detectors
  • On way descriptor speed up delay a week for object detection.
  • Spin image code sent, best way to match … maybe kdtrees
  • HOG training code
    • Put OpenCV extra
      • svn opencv.lib …/trunk-opencv-extra
    • Default coefficients are built in for pedestrian detection
  • Website for user contribution
    • Need voting site

FUTURE, POST RELEASE

  • Simple interface for object learning model
    • Optimize parameters by cross validation
    • Compare by ROC curve
      • False alarm vs False detect automated
      • prediction chart
      • variable importance
    • Cross validation
    • R interface (?) …
      • See if R interface was done Vasili (in outsourcing company) (done for PNL?)
    • “HighGUI for ML”
      • Brings up interactive window
        • Visualize model
        • test error
        • model complexity
        • In python (?)
  • More features

Vadim:

  1. C-Descriptor is 99% finished (the absent load/save functions will be added today). The code will be sent to Michael Calonder, who kindly agreed to review it.
    • The feature training part can do either random projection-based (as in the original code) or PCA-based (as Michael adviced me to do) compression of the output classifier.
  2. kMeans initialization has been changed (from random labels to random centers) for much better convergence properties. That fixed OpenCV SF bug #2804614.
  3. OpenCV has been successfully built on Win64 (except for ffopencv module that requires precompiled 64-bit ffmpeg libs on Windows). Many warnings have been fixed; probabilistic Hough transform has been modified to work around internal MS compiler error. All the tests except for DFT, DCT and addWeighted passed.
  4. Another bug in GEMM has been fixed (reported by Anatoly)
  5. 2 priority-6 bugs reported by James Bowman have been fixed.
  6. Most of the highgui API (with the notable exception of video I/O functonality) has been converted to C++.
    • 1 week, might be lower priority
  7. Maria finished debugging haartraining. On our face database the new version is capable of producing cascade classifiers comparable with the original cascades.
  8. Simple version of one-way descriptor compatible with the new OpenCV C++ data structures has been implemented by Anatoly (~80% complete). It is derived from the *Victor*’s code, but much shorter than the original because it’s lower-level (does not include any I/O or dataset management) and all the memory management is done automatically by C++. Another bug (basically the same bug with patch blurring) has been found and reported to Victor.

OpenCV release status:

  • I’m afraid, we need to postpone the code freeze by 1-2 more weeks.

Action Items

Gary

  • Contact James about unit tests in python, CC Vadim
  • Ask Kurt / James about Bundle adjustment in ROS (J.D Chen’s code)
  • Ask Scott about voting site for user contribution

Vadim

  • Talk to Sergey about bundle adjustment on his side

From Last Time

Vadim

  • Look at priority 6 python bugs
    • Will look at today

James

  • Merge doc2 to doc
  • In generated html
    • Document bug button(s)
      • OR: Other ways of getting feedback such as a comment sections
    • link back to OpenCV Wiki
    • cmake to build

Caroline

  • Email mask interface considerations

Done

Not done:

  • 3D features
  • Nistor Tree
  • Figure out code contribution process
  • Documentation bug link
  • Fast Chamfer with orientation

2009-06-23

Agenda

  • Status, go through summer release list below
  • Documentation, still no feedback button.
  • Bundle adjustment status?
  • HOG training
  • One way descriptor — Victor
  • User contribution
  • Post release …
    • Browser connections?
    • Simple mesh routines
    • 3D fitting
    • Of course — object recognition: features, routines

OpenCV Summer Release

  • Feature freeze end of May, June for test and doc.
  • General Structure:
    • :\ ? Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
  • New Features
    • C++ interface
      • {OK} cxcore
      • cv — start with filtering functions, resize, optical flow, object detection
  • Better python interface
    • In progress thanks to (James Bowman)
  • 2D features
    • {OK} HOG
      • Boundary Fragments … In progress thanks to Marius Muja
    • C-descriptor … (Vadim)
      • :\ ? See below for LePetit style randomized tree.
    • {OK} MSER
    • {OK} Star
    • Self-similar detector
  • {X} Alpha channel, alternatives to contours RLE
    • {X} New alpha transparency (per pixel) added
    • {X} Introduce functions with advance ways of blending
    • {X} Mask channels
  • 3D features
    • Bundle Adjustment (probable contribution by Phase Space)
    • Plane fit (Gary from Radu’s code)
    • LePetit style one way feature (Victor) (to Anatoly)
    • {X} PFH, FPFH from Radu,
    • {X} 3D Delaunay,
    • {X} 3D convex hull,
    • {X} mesh generation,
    • normals in grid (Gary)
    • normal based 3D patch (Gary)
    • {OK} SPIN
    • {X} 3D capture http://www.3dsom.com/index.html
    • {X} VSLAM
    • {X} Image stitching
  • Optical flow
    • {OK} Dense (Farnerback)
    • {X} Variational optical flow
  • MLL
    • {OK} Extreme trees
    • {OK} Reduce memory footprint
    • {OK} More convenient training via a data class
    • Nister trees (Patrick)
    • FLANN or appropriate parts (Marius)
    • :\ ? LePetit style random trees
    • Generalized (boosted) cascade (generalizing Haar cascade)
  • Misc:
    • {X} Focus detector,
    • {OK} adaptive skin tracking
  • Optimizations
    • Various functions, ongoing …
  • Documentation
    • Latex
  • High GUI
    • In memory .jpg, .png
    • {X} move to C++ interface
  • Testing functions
    • Regression tests for all functions
  • Define user contribution process:

Minutes

  • NEED TO MAKE SURE PEOPLE GET CREDIT — will put in contributors page in wiki
    • http://opencvlibrary.svn.sourceforge.net/svnroot/opencvlibrary/trunk/opencv/THANKS
  • Bugs in C-descriptor, look into and compare against ROS implemenation
  • Implemented kd-tree based on Xavier code, see below
  • K-means+algorithms by Aurthur and Vassilvitskii have better method of choosing centers.
    • First center at random
    • Other centers from a distribution that biases towards distant centers
    • thereafter regular kmeans
    • Fixed reported bug — takes many less iterations to converge
      • Might be in FLANN
  • Integration of FLANN
  • Gary introduced more direct link to bugs from wiki
  • Bundle adjustment API
    • Sergey implementing functionality with agreed API
  • OpenCV works fine under Mac OS
  • HOG training code.
    • Try out original training code
      • Original code might have bugs in it. Binaries work for Mac, but maybe not source code for Linux
      • They use linear SVM, say RBF works but slower
      • Maybe upload to opencv extras and point to it.
  • One way descriptor
    • Not waiting on Victor for now, but code needs this week for Beta release
    • One way descriptor compares PCA affine perturbed patches
    • C-descriptors compares PCA signatures from random tree affine perturbed patches.
      • Once and for all alphabet
    • One way descriptors can use as many features as want, can keep adding
  • 3D capture, Anatoly is experimenting with keypoint detectors and descriptors. Tried FAST last week.
    • Good features to track is faster than FAST and doesn’t seem worse
  • John Hsu is using 3D models in simulator for training and test — good way to go
    • Have more and more capture techniques
    • Silhouette capture
  • Still no decision about user contribution
  • Nistor trees

Vadim

  1. C-Descriptor is complete, but the results are pretty bad (very few found corresponces are correct). I continue debugging the code.
  2. New C++ version of kd-trees, based on Xavier Delacour code, has been created. It is ~20% more accurate (in some cases ~2x more accurate) than the previous version (because of better prunning of the tree branches) and yet significantly faster, especially the tree construction part and the query in case of low dimensionality. Currently it only supports L2 metrics, but L1 is quite easy to add.
  3. Another impovement in KMeans: now it’s possible to use the kmeans++ algorithm by David Arthur and Sergei Vassilvitskii, which is theoretically and empirically much superior to the random center initialization.
  4. Maria fixed couple more bugs in haartraining and started integration of LBP (local binary pattern) features, which, in one hand, is close enough to Haar features and, in the other hand, could be a good verification that haartraining is extensible.
  5. Talked to Sergey and agreed on the bundle adjustment API.
  6. Added precompiled header support for Xcode and GNU makefiles, which should reduce build time.

Plans for the week:

  1. finish debugging cdescriptor.
  2. integrate one-way descriptor.
  3. replace the old haartraining with the new one.

Action Items

Gary

  • Ask about Nistor Trees
  • Have Marius contribute Chamfer+orientation
  • Talk to Marius about contributing FLANN
  • PUT IN CONTRIBUTORS DIRECTORY INTO WIKI
  • Talk with Ian Goodfellow about bugs in HOG training code
  • See if we can get Balan’s matlab code for symmetric object 3D capture

Vadim

  • Find out about user contribution website similar to Matlab style

PREVIOUSLY

Gary

  • Contact James about unit tests in python, CC Vadim
  • Ask Kurt / James about Bundle adjustment in ROS (J.D Chen’s code)
  • Ask Scott about voting site for user contribution

Vadim

  • Talk to Sergey about bundle adjustment on his side

Done

Not done:

  • 3D features
  • Nistor Tree
  • Figure out code contribution process
  • Fast Chamfer with orientation

2009-07-02

Agenda

  • Status, code/freeze. go through summer release list below
    • Beta release
  • FLANN from/by Marius
    • *Marius*’s Chamfer matching
  • High GUI interface
  • New descriptors from bazAR tracking package
  • Bundle adjustment status?
  • HOG training
  • One way descriptor — Victor
  • User contribution
  • Post Release
    • Alex Teichman’s feature framework
      • His “kernel based” boosting framework
    • Optimized pipelines

OpenCV Summer Release

  • Feature freeze end of … sometime in July for test and doc.
  • Release in August, 2009
  • General Structure:
    • :\ ? Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
  • New Features
    • {OK} C++ interface
      • {OK} cxcore
      • {OK} cv
      • {OK} HighGUI (except for video I/O
    • Better python interface
      • {OK} In progress thanks to (James Bowman)
      • (./) Use this to expand unit test functions for OpenCV
  • 2D features
    • {OK} HOG
      • (./) Chamfer matching with orientation In progress thanks to Marius Muja
    • (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
      • (./) LePetit keypoint detector YAPE (Yet Another Point Extractor). We have found this to be the most stable detector.
      • {OK} C-descriptor … (Vadim)
      • (./) LePetit one way descriptor (Victor) (to Anatoly)
    • {OK} MSER
    • {OK} Star
    • (./) Self-similar detector
  • {X} Alpha channel, alternatives to contours RLE
    • {X} New alpha transparency (per pixel) added
    • {X} Introduce functions with advance ways of blending
    • {X} Mask channels
  • 3D features
    • (./) Bundle Adjustment (probable contribution by Phase Space)
    • {X} Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
    • {X} PFH, FPFH from Radu,
    • {X} 3D Delaunay,
    • {X} 3D convex hull,
    • {X} mesh generation,
    • (./) normals in grid (Gary)
    • (./) normal based 3D patch (Gary)
    • {OK} SPIN
    • {X} 3D capture http://www.3dsom.com/index.html
    • {X} VSLAM
    • {X} Image stitching
  • Optical flow
    • {OK} Dense (Farnerback)
    • {X} Variational optical flow
  • MLL
    • {OK} Extreme trees
    • {OK} Reduce memory footprint
    • {OK} More convenient training via a data class
    • (./) Nister trees (Patrick)
    • (./) FLANN or appropriate parts (Marius)
    • {X} LePetit style random trees
    • {OK} Generalized (boosted) cascade (generalizing Haar cascade)
  • Misc:
    • {X} Focus detector,
    • {OK} adaptive skin tracking
  • Optimizations
    • (./) Various functions, ongoing …
  • Documentation
    • {OK} Move to one Latex source
      • {OK} Use to generate HTML, PDF
    • {OK} Add search function
    • {OK} Add user feedback/documentation bug system
    • (./) Proof read
    • (./) Add content
    • (./) Create C++ documentation
    • (./) Create Python documenation
  • High GUI
    • {OK} In memory .jpg, .png
    • {OK} move to C++ interface
    • {X} move to using QT as the HighGUI interface look and feel
  • Testing functions
    • (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
  • (./) Define user contribution process:

Minutes

Vadim

  1. c-descriptor has been debugged (thanks to Rainer for the code review and useful recommendations on what could be wrong). It now shows comparable with SURF localization accuracy, provided that the object scale in the scene is specified more or less correctly. Much better robustness can be achieved by using Lepetit’s YAPE (Yet Another Point Extractor) detector – see item 2 below. Perhaps, it makes sense to make more universal class than just C-descriptor. The class can easily combine randomized trees-based classifier (as in the original paper by Lepetit from 2006), Michael Calonder’s c-descriptor, ferns-based (a.k.a. semi-naive Bayesian) classifier and the corresponding descriptor. The techniques used are very similar, and in the papers ferns are claimed to provide better accuracy than randomized trees.
  2. We did some experiments with open-source (GPL) bazAR tracking package from cvlab.epfl.ch site and the results are very promising – the keypoint detector constantly finds the same points on the quite different views of the same object, i.e. the stability is great! Unfortunately, we could not make the multi-scale variant of the detector work, but the idea is clear, and, I think, we can do independent implementation pretty quickly.
  3. Maria added LBP features to the new “traincascade” application (the substitute for haartraining). It worked well and now we are adding the corresponding detection function to OpenCV (for now it will be sort of reference code w/o any serious optimization).
  4. Generation of normally distributed random numbers is now ~3x faster with Ziggurat algorithm (it can be useful for faster keypoint detection and classifier training using Lepetit method, as they add uniform or gaussian noise to the generated patches (and seem to prefer the gaussian noise))
  5. Several bugs have been fixed: 1) build issues on Linux when using autotools, 2) precompiled headers did not work well on Linux, 3) crash in calcHist (reported by Alex Teichman), 4) crash in mltest, found and fixed by Maria

Discussion Minutes

  • Stable keypoint detection. Lepetit’s group
    • Run keyboard detector, used pyramid, maximal Laplacian detection with edge suppression
      • Repeat procedure 100-1000 times using affine distortion (know where the correspondences are)
      • Find most stable keypoints that are stable to affine distortion
      • Train classifiers using forest or ferns
        • Can be GPU’d (Anatoly knows for rendering) OpenCL Intel/NVidia, CPI, AMD
        • Suitable for SSE for image warping
      • One way descriptor uses multiple points in kd-tree, others use patches in random forests/ferns
      • In bazAR there are several demos, keypoint detection and training in randomized trees
    • We have c-descriptor code but bazAR performs better, maybe it will work the same with the better key points above
    • Will finish and commit to SVN this week
    • C descriptors are in
  • FLANN
    • Marius have commit privilege
    • In 3rd party directory
      • Documentation to be melded in …
  • Alex Tiechman sent in feature pipeline for architectural feedback
    • We may consider GPU of high payoff features found with this
  • HighGUI
    • Probably use QT (LGPL) ~GTK
    • Do it after release
  • Bundle Adjustment
    • Sergey said that it will be ready this Friday. Debugging
    • Initialization might not be included, just pure optimization
      • Will include unit tests
    • To run, you need to provide initial estimates for camera intrinsic and code
      • Documentation will describe
    • For complete 3D pipeline we are missing
      • Initialization
      • Bundle adjustment
      • Verification and
      • Mesh reconstruction
      • Probably only bundle adjustment will go into Summer release. Anatoly working on
        • Waits on the robust keypoints from LePetit’s group
        • LePetit’s group uses larger solution ImageModeler (Autodesk product)
  • HOG
    • Put the training code for now to OpenCV extra
    • Coefficients for human detection are shipped
  • Python
    • If interface is C++
    • We can adjust James scripts to handle new functions
      • New functions create output images will be created or resized if haven’t been created before
  • For BETA, we want
    • C-descriptor
    • LePetit’s affine robust YAPE keypoint detector
    • improved training cascade classifier (training cascade supporting haar and other featuers
    • Self-similarity feature
    • Brief documentation for C++ interface
    • ~ FLANN Integration
    • Bundle adjustment
  • For Release
    • Make Python interface for C++ functionality
    • FLANN
    • Finish documentation
    • Write python tests
    • Chamfer matching
    • One way descriptor

Action Items

Gary

  • Ask Marius about chamfer matching with orientation
    • Documentation ? LaTex into OpenCV
  • Look into purchasing ImageModeler
  • Talk with James about interface to C++ code

PREVIOUSLY

Gary

  • {X} Ask about Nistor Trees
  • :\ Have Marius contribute Chamfer+orientation
  • (./) Talk to Marius about contributing FLANN
  • {X} PUT IN CONTRIBUTORS DIRECTORY INTO WIKI
  • :\ Talk with Ian Goodfellow about bugs in HOG training code
  • {X} See if we can get Balan’s matlab code for symmetric object 3D capture
  • {X} Ask Scott about voting site for user contribution

Vadim

  • Find out about user contribution website similar to Matlab style

Not done:

  • 3D features
  • Nistor Tree
  • Figure out code contribution process
  • Fast Chamfer with orientation

2009-07-06

Agenda

  • Static build for OpenCV or other minimal footprint methods?
  • Cascade generalization Haar | LBP or Haar & LBP ?
    • New facial features working with new cascade?
  • Bundle adjustment status
  • User contribution / website with voting
  • Documentation read-over.
    • Grammar being corrected now, almost done
  • Victor: Status on outlet detectors
  • For BETA, we want
    • {OK} C-descriptor
    • {OK} LePetit’s affine robust YAPE keypoint detector
      • They use 1 way descptor, C-descriptor
    • {OK} improved training cascade classifier (training cascade supporting haar and other featuers
    • Self-similarity feature
    • Brief documentation for C++ interface
    • (./) FLANN Integration
    • Bundle adjustment
  • For Release
    • Make Python interface for C++ functionality
    • FLANN
    • Finish documentation
    • Write python tests
    • Chamfer matching
    • One way descriptor
  • Nistor trees (ask Patrick)
  • Python interface for C++ API — what needs to be done (Ask James)
  • Status of ML documentation (ask James)
  • Chamfer matching code (ask Marius)
  • FLANN (Marius)
    • Does Marius have commit privileges yet?

OpenCV Summer Release

  • Feature freeze end of … sometime in July for test and doc.
  • Release in August, 2009
  • General Structure:
    • :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
  • New Features
    • {OK} C++ interface
      • {OK} cxcore
      • {OK} cv
      • {OK} HighGUI (except for video I/O
    • Better python interface
      • {OK} In progress thanks to James Bowman
      • (./) Use this to expand unit test functions for OpenCV
  • 2D features
    • {OK} HOG
      • (./) Chamfer matching with orientation In progress thanks to Marius Muja
    • (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
      • (./) LePetit keypoint detector YAPE (Yet Another Point Extractor). We have found this to be the most stable detector.
      • {OK} C-descriptor … (Vadim)
      • (./) LePetit one way descriptor (Victor) (to Anatoly)
    • {OK} MSER
    • {OK} Star
    • (./) Self-similar detector
  • {X} Alpha channel, alternatives to contours RLE
    • {X} New alpha transparency (per pixel) added
    • {X} Introduce functions with advance ways of blending
    • {X} Mask channels
  • 3D features
    • (./) Bundle Adjustment (probable contribution by Phase Space)
    • {X} Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
    • {X} PFH, FPFH from Radu,
    • {X} 3D Delaunay,
    • {X} 3D convex hull,
    • {X} mesh generation,
    • (./) normals in grid (Gary)
    • (./) normal based 3D patch (Gary)
    • {OK} SPIN
    • {X} 3D capture http://www.3dsom.com/index.html
    • {X} VSLAM
    • {X} Image stitching
  • Optical flow
    • {OK} Dense (Farnerback)
    • {X} Variational optical flow
  • MLL
    • {OK} Extreme trees
    • {OK} Reduce memory footprint
    • {OK} More convenient training via a data class
    • (./) Nister trees (Patrick)
    • (./) FLANN or appropriate parts (Marius)
    • {X} LePetit style random trees
    • {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)
  • Misc:
    • {X} Focus detector,
    • {OK} adaptive skin tracking
  • Optimizations
    • (./) Various functions, ongoing …
  • Documentation
    • {OK} Move to one Latex source
      • {OK} Use to generate HTML, PDF
    • {OK} Add search function
    • {OK} Add user feedback/documentation bug system
    • (./) Proof read
    • (./) Add content
    • (./) Create C++ documentation
    • (./) Create Python documenation
  • High GUI
    • {OK} In memory .jpg, .png
    • {OK} move to C++ interface
    • {X} move to using QT as the HighGUI interface look and feel
  • Testing functions
    • (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
  • (./) Define user contribution process:

Minutes

  • Static builds of the library will be done past Beta release. Need to see if it is easy.
  • Cannot mix features in one cascade
  • New Cascade application still supports old formats
  • Bundle adjustment …
    • Will talk to Sergey next week.
      • Debugging the code, no estimate yet on completion
  • Still don’t know about a voting website for user contributed code. Ask at Argus
    • Matlab exchange as “prototype”
    • Need to develop a simple template contribution function that goes all the way through
  • _’FLANN_’
    • Has public interface for the functionality — wrap those in OpenCV structures
      • Example that Vadim proposed
      • Write regression tests for it
      • Marius needs write permission.
        • Needs some feedback
        • Marius to send in sourceforge log in
          • Start today — probably this week done
  • Chamfer matching
    • Currently a library in ROS, depends on OpenCV only
    • Interface needs cleaning
      • Chamfer match over entire image
      • Or use list of starting points
      • Has orientation but maybe need orientation flag to turn it on and off
        • Marius put in orientation on or off flag
        • When ready, send email to Vadim
  • Testing c-descriptor with YAPE detector
  • One way descriptor, couple of days, but delayed to next week
  • Outlet detection
    • Fast version of one way descriptor is written
      • Changing the interface
      • Talked to author Stefan Hinterstoisser
        • Stephen might share the code
      • 2 days for restructuring the interface
    • Implemented Generalized Hough Transform
      • 2 stages, 1st to filter out outliers, 2nd to refine detection
      • 1 week
    • Have a lot of data from forearm
      • Could generalize the orange outlet code
      • End of the week decide
    • Move to test on forearm with one way.

Reports:

Vadim

  1. Lepetit’s keypoint detector is finished, now it’s being tested together with the c-descriptor.
  2. Maria finished the detection function for LBP features. This week the new traincascade application will be integrated to OpenCV.
  3. Several build issues have been resolved (reported by JD, Oliver Sidla an other guys). Also, it’s now possible to include top-level OpenCV CMakeLists.txt into a bigger project – which is very convenient.
  4. Added haar cascades for facial feature detection (contributed by Modesto Castrillon Santana).

Action Items

Gary

  • Ask Marius about chamfer matching with orientation
  • Talk with James about python interface to C++ code
  • Ask about Nistor Trees
  • Chamfer code
  • PUT IN CONTRIBUTORS DIRECTORY INTO WIKI
  • Ask Scott about voting site for user contribution
    • Think of some template application as an example
  • Documentation “gaps” read over

Vadim

  • Find out about user contribution website similar to Matlab style

PREVIOUSLY

Gary

  • (./) Look into purchasing ImageModeler — trying to get download for 3dsom instead
  • {./} See if we can get Balan’s matlab code for symmetric object 3D capture

Vadim

  • :\ Find out about user contribution website similar to Matlab style

2009-07-14

Agenda

  • Release progress review
  • Contribution site … blah
  • Outlet summary
  • One way descriptor news?
    • Stefan Hinterstoisser release code?
  • Bundle adjustment status
  • YAPE
  • Static builds?
  • 3D feature status
    • I have Matlab code for symmetric object model creation.
  • C-descriptor in?
  • FLANN and 2D Chamfer
  • ’Post release
    • Flesh detection techniques

OpenCV Summer Release

  • Feature freeze end of … sometime in July for test and doc.
  • Release in August, 2009
  • General Structure:
    • :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
  • New Features
    • {OK} C++ interface
      • {OK} cxcore
      • {OK} cv
      • {OK} HighGUI (except for video I/O
    • Better python interface
      • {OK} In progress thanks to (James Bowman)
      • (./) Use this to expand unit test functions for OpenCV
    • 2D features
      • {OK} HOG
        • (./) Chamfer matching with orientation In progress thanks to Marius Muja
      • (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
        • (./) LePetit keypoint detector YAPE (Yet Another Point Extractor). We have found this to be the most stable detector.
        • {OK} C-descriptor … (Vadim)
        • (./) LePetit one way descriptor (Victor) (to Anatoly)
      • {OK} MSER
      • {OK} Star
      • (./) Self-similar detector
    • {X} Alpha channel, alternatives to contours RLE
      • {X} New alpha transparency (per pixel) added
      • {X} Introduce functions with advance ways of blending
      • {X} Mask channels
    • 3D features
      • (./) Bundle Adjustment (probable contribution by Phase Space)
      • {X} Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
      • {X} PFH, FPFH from Radu,
      • {X} 3D Delaunay,
      • {X} 3D convex hull,
      • {X} mesh generation,
      • (./) normals in grid (Gary)
      • (./) normal based 3D patch (Gary)
      • {OK} SPIN
      • {X} 3D capture http://www.3dsom.com/index.html
      • {X} VSLAM
      • {X} Image stitching
    • Optical flow
      • {OK} Dense (Farnerback)
      • {X} Variational optical flow
    • MLL
      • {OK} Extreme trees
      • {OK} Reduce memory footprint
      • {OK} More convenient training via a data class
      • (./) Nister trees (Patrick)
      • (./) FLANN or appropriate parts (Marius)
      • {X} LePetit style random trees
      • {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)
  • Misc:
    • {X} Focus detector,
    • {OK} adaptive skin tracking
  • Optimizations
    • (./) Various functions, ongoing …
  • Documentation
    • {OK} Move to one Latex source
      • {OK} Use to generate HTML, PDF
    • {OK} Add search function
    • {OK} Add user feedback/documentation bug system
    • (./) Proof read
    • (./) Add content
    • (./) Create C++ documentation
    • (./) Create Python documenation
  • High GUI
    • {OK} In memory .jpg, .png
    • {OK} move to C++ interface
    • {X} move to using QT as the HighGUI interface look and feel
  • Testing functions
    • (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
  • (./) Define user contribution process:

Minutes

Planar detections

  • Use ferns.
    • Entry is computed in large table. Set of responses of every binary weak classifiers give indices in a table
    • Learns that patch and tracks it
    • Can add c-descriptor functionality where we get descriptor key
  • YAPE keypoint is in separate class

Object detection

  • Cascasde needed a stand alone function for LBP
  • Adding new features — need to make it easier to add.
    • Add data structure
    • Load classifier from xml to memory
    • Load features
    • Call back that computes features for training and recognition
      • feature I/O
  • Maria working on avoiding code duplication as much as possible
    Todo
  • self similar features from Rainer
  • Cascade from Maria
  • Bundle adjustment integration
  • Will work on this week
  • Look at static builds

Bundle adjustment

  • Sergey was supposed to finish this week

3D

  • Almost done with my “surface plugs”
  • Do as the first contribution example
  • Spin images (Anatoly can concentrate on this)
    • Generation is done from point cloud to spin image
      • Parameter issues are fixed
    • Can’t use very well since you can only do pairwise matching
    • Need to filter geometrically
  • Anatoly tested 3dsom
    • Need good lighting
    • Good green screen
    • Don’t have sophisticated background segmentation
    • They use special pattern for camera pose ID
    • Capture object from various views
    • Pass view to objects
    • tried on 2 or 3 objects and results were good with good lighting
    • Practiced capture takes less than half an hour per model. Maybe can be done in 10 minutes.

Documentation

  • Python (James)
  • C++ interface
  • Collect ctypes in one place, hyper link to it

One way descriptor

  • Later this week
  • No promise about sharing code

Outlets

  • Tomorrow will turn over and focus on one way
  • Orange outlets are still being tested
  • Need to do for robot
    • Now getting high percentage on some outlets >95%, low on others <95%
    • Much larger scale change in forearm, influences one way descriptor matching
    • Camera position estimation needs to be redone for lots of different outlets
      • 2×1 with same geometry
      • 2×2 with same geometry
      • Several views of outlets, assuming distance of holes are same, can use to refine scale/structure from motion
    • With orange outlets we had only one
      • The robot should be able to launch specific training (single frontal view with holes marked)
        • Mech turk holes
    • Hope by end of the week to have start of end to end

Toolkits

  • 3D model/mesh capture
    • symmetric objects
  • Feature framework Tiechman working on
  • Color space handling/invarient color
    • Flesh detection
      • Histogram
      • Adaptive skin ?

Reports:

Victor

Outlet detection:

One way descriptor code has been revised. CvOneWayDescriptor code has been reviewed and is ready for submission to OpenCV. The helper class CvOneWayDescriptorBase that does actual work on comparing and classifying the descriptors is under revision now. The fast version of the feature generation has been implemented based on precomputed descriptors of pca components of features. The new code is faster than the previous one at least one order of magnitude (generating descriptors for 100 features takes several seconds instead of several minutes now).

The generalized hough transform for detecting outlets has been implemented and tested on the forearm data. Its integration into the outlet detection ROS package is on the way. The summary statistics for forearm data is given below.

Outlet Detection Forearm Camera
|Outlet_|Num Images_|Num Detected_|
|cracked |100 |79 |
|fake_wall_white |113 |111 |
|fake_wall_orange_2x2 |35 |35 |
|orange_2x1 |103 |88 |
|conference_white_2x2_extreme_lighting |67 |20 |
|hall
white
2×1 |103 |100 |
|white
2×2 |101 |70 |

Spin images:

Anatoly has started improving spin images code. Random sampling of spin images, automatic generation of spin image parameters have been implemented. Spin image correspondence filtering based on geometric consistency is in progress.

Vadim

  1. Finally, planar object detection is working! It involves Lepetit’s keypoint detector, fern-based point classifier (random trees can easily be added too), and RANSAC-based homography estimation. The results on the standard OpenCV sample is attached.
  2. Maria has implemented a draft version of standalone OpenCV object detection function based on LBP features. Some refactoring of the training application was needed to extract the code for reading classifier and running it, so that OpenCV libs do not depend on the whole source of “traincascade”.
  3. 8 bugs have been fixed, including 5 reported via SF bug tracker.

Action Items

Gary

  • Foward mesh improvement paper

Vadim

  • Mail Michael how to turn Fern into descriptors (log vs regular output).

PREVIOUSLY

Gary

  • (./) Ask Marius about chamfer matching with orientation
  • (./) Talk with James about python interface to C++ code
  • (./) Ask about Nistor Trees
  • (./) Chamfer code
  • {X} PUT IN CONTRIBUTORS DIRECTORY INTO WIKI
  • {X} Ask Scott about voting site for user contribution
    • Think of some template application as an example
  • {X} Documentation “gaps” read over
  • {./} See if we can get Balan’s matlab code for symmetric object 3D capture

Vadim

  • Find out about user contribution website similar to Matlab style

2009-07-21

Agenda

  • Release review
    • Documentation lag, though Machine learning docs now up.
  • Vincent LePetit is working for 2 weeks at Willow. Questions for him on YAPE or other descriptors?
    • Status of YAPE,
    • {OK} c-descriptor now in
  • Self similar feature?
  • Bundle adjustment status?
  • Static builds
  • 3D surface plugs by Thrusday.
    • Spin image status?
  • 3D capture status?
  • Alex Teichman’s “feature engine”?
  • Outlet detector to forearm status

. OpenCV Summer Release

  • Feature freeze end of … sometime in July for test and doc.
  • Release in August, 2009
  • General Structure:
    • :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
  • New Features
    • {OK} C++ interface
      • {OK} cxcore
      • {OK} cv
      • {OK} HighGUI (except for video I/O
    • Better python interface
      • {OK} In progress thanks to (James Bowman)
      • (./) Move into OpenCV and deprecate SWIG only version
      • (./) Use this to expand unit test functions for OpenCV
    • 2D features
      • {OK} HOG
        • (./) Chamfer matching with orientation In progress thanks to Marius Muja
      • (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
        • (./) LePetit keypoint detector YAPE (Yet Another Point Extractor). We have found this to be the most stable detector.
        • {OK} C-descriptor … (Vadim)
        • (./) LePetit one way descriptor (Victor) (to Anatoly)
      • {OK} MSER
      • {OK} Star
      • {OK} Self-similar detector
    • {X} Alpha channel, alternatives to contours RLE
      • {X} New alpha transparency (per pixel) added
      • {X} Introduce functions with advance ways of blending
      • {X} Mask channels
    • 3D features
      • (./) Bundle Adjustment (probable contribution by Phase Space)
      • {X} Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
      • {X} PFH, FPFH from Radu,
      • {X} 3D Delaunay,
      • {X} 3D convex hull,
      • {X} mesh generation,
      • (./) normals in grid (Gary)
      • (./) normal based 3D patch (Gary)
      • {OK} SPIN
      • {X} 3D capture http://www.3dsom.com/index.html
      • {X} VSLAM
      • {X} Image stitching
    • Optical flow
      • {OK} Dense (Farnerback)
      • {X} Variational optical flow
    • MLL
      • {OK} Extreme trees
      • {OK} Reduce memory footprint
      • {OK} More convenient training via a data class
      • (./) Nister trees (Patrick)
      • (./) FLANN or appropriate parts (Marius)
      • {X} LePetit style random trees
      • {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)
  • Misc:
    • {X} Focus detector,
    • {OK} adaptive skin tracking
  • Optimizations
    • (./) Various functions, ongoing …
  • Documentation
    • {OK} Move to one Latex source
      • {OK} Use to generate HTML, PDF
    • {OK} Add search function
    • {OK} Add user feedback/documentation bug system
    • {OK} Proof read
    • {OK} Put up
      • {OK} cxcore
      • {OK} cv
      • {OK} MLL
      • {OK} Aux
      • {OK} HighGUI
    • (./) Add content
    • (./) Create C++ documentation
    • {OK} Create Python documenation
  • High GUI
    • {OK} In memory .jpg, .png
    • {OK} move to C++ interface
    • {X} move to using QT as the HighGUI interface look and feel
  • Testing functions
    • (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
  • (./) Define user contribution process:

Minutes

Updating latest tested snapshot for ROS (cmake 2.4.6 or .7)

  • Tomorrow (Wed or Thrus)
  • Some versions of cmake don’t handle everything we need. Between 2.4-2.6 some version doesn’t make 2.6.0 — need to check in Ubuntu 8.10

3D capture

  • Just looked at the paper
  • But Anatoly busy with spin image

Spin image

  • Changing the inteface
  • Adding more features
  • Adding correspondence parts, filtering out wrong correspondences based on consistency
    • Matching spin images 1 by 1 is too ambiguous
  • 3D recognition project
  • Couple more weeks

User contribution

  • Asked local designer
    • He’s asking around
    • Waiting

Static builds

  • Not sure if before beta or after
    • Needs testing
    • cmake makes it easy, just put “static” instead of “shared”
      • But question of how to use them properly
      • Might build cmake option to build tests section statically
    • After Beta

Signature based on Ferns questions

  • multiplying signatures (or summing their logarithms) tends to be better
    • For ferns
  • For the signatures, better to do sums
    • Michael Calonder used random trees
      • Computed PCA compressed probability signatures
      • Detection stage, these were summed
      • Recent paper showed multiplied these, PAMI 2009
    • For signatures, use the sum of the probabilities
      • To compare, used simple NN search. K-d trees, Vocab trees
  • Point detector YAPE
    • Vadim tried both versions
      • Ferns version is latest
      • Previous found more robust to affine
    • compute LoG but diagonal edges can cause false features due to aliasing
    • Tried FAST detector — was slower than expected
      • Compared Good features to track, SURF, found first version of YAPE
    • Written from scratch version of older YAPE is in OpenCV
    • Speed of YAPE vs Fast — haven’t measured yet, but think they are comparable, maybe YAPE is faster.

Vadim is done with his code, waiting for other adjustments

  • Waiting for Cascade
  • Waiting for Bundle adjustment

Bundle adjustment

  • Will talk to Sergey tomorrow with separate mail about it

One way descriptor

  • Basically runs
  • But need a more general class, right now its tied to outlets
  • By Friday

Cascade

  • Detection
    • You write a small class that reads the xml feature during load time
  • Training
    • Need to write xml features to file and
    • enumerate all features
    • After complete, the whole cascade is written to an xml file.
      • First part is indepenedent of features — the cascade and params
      • Then actual features
    • For haar, 2 or 3 rectangles

Reports:

Marius has been working on

  • FLANN and
  • Chamfer
  • Neighther commited now

Vadim

The following has been done last week:

  1. Self-similarity descriptor has been integrated to OpenCV (it’s in SVN trunk now)
  2. More than 20 bug tracker records have been closed.
  3. While not in the tracker, lot’s of fixes have been made in C++ sparse matrices (as Victor and Alexey are looking at making use of it).
  4. The latest videoInput & ffmpeg 0.5.0 have been added to the repository; they are by highgui on Windows.
  5. Maria finished implementation of standalone OpenCV C++ class for Haar features and started implementation of similar class for LBP features.
  6. Anatoly continues work on spin image improvement. A whole big C++ class for a set of spin images representing a 3D model has been designed and implemented.

There is also helper cv::Mesh3D class that can only store points, octtree and normals, but could contain some triangulation information in the future.

Victor

  • The code for working with new outlet detection is in repository. In order to use it you have to define macro GHT and recompile outlet.dylib. The interface is the same, however you have to explicitly load a template for each outlet you are looking for. An example is in sample.cpp, line 75. You can run the code on forearm data by starting outletsample binary and passing 5 parameters to it (second to last being the path to the outlet template).
    • Creating a template is getting a single image and then editing a YML file.
      • No instructions so far
      • Make and image
      • Crop it to only outlet
      • Label the holes
      • Copy the rest from existing templates
      • One way descriptor is many images affine transformed
        • 20 PCA components
        • Create affine versions
        • Recombined — 100M for this application (once and for all for any US type outlets)
          • User just has to create a template for each outlet
    • Will make a simplified tool
    • Add to Cmake.lists need to add _GHT there
  • I will send you now edited forearm images (collected by Patrick and myself and rearranged in convenient form).
  • I will also send you the outlet templates. This will be a zip file containing two folders — configs (containing the templates themselves) and descriptors (containing precomputed highres and lowres pca components as well as the precomputed one way descriptors of highres pca components). The templates are in configs folder, for instance, in order to detect fake wall orange 2×2 outlet, use configs/forearm/fake_wall_2x2orange. Templates reference descriptors with relative path so configs and descriptors folders should always be in the same folder.
  • The code works now 0.5-1.5sec (depending on an outlet) on my laptop. We are working to make it work faster and more accurately. We didn’t investigate what would happen if a template for one outlet is used for detecting another, but on earlier stage of this project I was experimenting with two different white 2×1 outlets and the matching results were bad so I think for now a specific template should be used for each outlet.
  • Outlet detection: the GHT code has been speeded up a factor of 6 by more efficient memory managing and replacing high-resolution hough histogram with a low-resolution followed by post-processing. The accuracy remains roughly the same, the processing time is 0.5-1.5s per frame. More speedups are on the way.
  • The code has been restructured to allow simpler use. The format of templates has been simplified.
  • Camera pose estimation for arbitrary outlets has been implemented.
  • Accuracy is >90%. Extreme lighting has 30% error, but in this set, one out of 4 outlets is very saturated, so perfect results are not possible.

Action Items

Gary

  • Ask Brian about storing large data (outlet specific data — image plus hole data and then PCA components (small) and descriptors (large 100M uncompressed)

Vadim

  • Send reminder email to web designer about user contribution site

Victor

  • Add one way descriptor.

PREVIOUSLY

Gary

  • (./) Foward mesh improvement paper
  • {X} PUT IN CONTRIBUTORS DIRECTORY INTO WIKI
  • {X} Ask Scott about voting site for user contribution
  • {X} Documentation “gaps” read over
  • Ask James about python documentation

Vadim

  • (./) Mail Michael how to turn Fern into descriptors (log vs regular output).
  • (./) Find out about user contribution website similar to Matlab style

2009-07-29

Agenda

  • Outlet
    • data, where to store
    • Plans — 8mm lens
    • Arms
  • One way descriptor
  • User contribution
  • Daisy code is available
  • {OK} Self similar feature
  • Bundle adjustment
  • Spin images
  • Documentation
  • Merging python in
  • FLANN, Chamfer, Nister
  • Post release
    • Static builds
    • 3D Capture
    • Alex’s feature engine

. OpenCV Summer Release

  • Feature freeze end of … sometime in July for test and doc.
  • Release in August, 2009
  • General Structure:
    • :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
  • New Features
    • {OK} C++ interface
      • {OK} cxcore
      • {OK} cv
      • {OK} HighGUI (except for video I/O
    • Better python interface
      • {OK} In progress thanks to (James Bowman)
      • (./) Move into OpenCV and deprecate SWIG only version
      • (./) Use this to expand unit test functions for OpenCV
    • 2D features
      • {OK} HOG
        • (./) Chamfer matching with orientation In progress thanks to Marius Muja
      • (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
        • (./) LePetit keypoint detector YAPE (Yet Another Point Extractor). We have found this to be the most stable detector.
        • {OK} C-descriptor … (Vadim)
        • (./) LePetit one way descriptor (Victor) (to Anatoly)
      • {OK} MSER
      • {OK} Star
      • {OK} Self-similar detector
    • {X} Alpha channel, alternatives to contours RLE
      • {X} New alpha transparency (per pixel) added
      • {X} Introduce functions with advance ways of blending
      • {X} Mask channels
    • 3D features
      • (./) Bundle Adjustment (probable contribution by Phase Space)
      • {X} Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
      • {X} PFH, FPFH from Radu,
      • {X} 3D Delaunay,
      • {X} 3D convex hull,
      • {X} mesh generation,
      • (./) normals in grid (Gary)
      • (./) normal based 3D patch (Gary)
      • {OK} SPIN
      • {X} 3D capture http://www.3dsom.com/index.html
      • {X} VSLAM
      • {X} Image stitching
    • Optical flow
      • {OK} Dense (Farnerback)
      • {X} Variational optical flow
    • MLL
      • {OK} Extreme trees
      • {OK} Reduce memory footprint
      • {OK} More convenient training via a data class
      • (./) Nister trees (Patrick)
      • (./) FLANN or appropriate parts (Marius)
      • {X} LePetit style random trees
      • {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)
  • Misc:
    • {X} Focus detector,
    • {OK} adaptive skin tracking
  • Optimizations
    • (./) Various functions, ongoing …
  • Documentation
    • {OK} Move to one Latex source
      • {OK} Use to generate HTML, PDF
    • {OK} Add search function
    • {OK} Add user feedback/documentation bug system
    • {OK} Proof read
    • {OK} Put up
      • {OK} cxcore
      • {OK} cv
      • {OK} MLL
      • {OK} Aux
      • {OK} HighGUI
    • (./) Add content
    • (./) Create C++ documentation
    • {OK} Create Python documenation
  • High GUI
    • {OK} In memory .jpg, .png
    • {OK} move to C++ interface
    • {X} move to using QT as the HighGUI interface look and feel
  • Testing functions
    • (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
  • (./) Define user contribution process:

Minutes

  • Outlet detectors
    • Multi-scale outlet detector is getting ~100% for high scale variation dataset
    • Specular dataset is only one that has lots of failures
    • Wrote outlet data labeler
    • Made template creation application
      • template creation
      • data set labeling
      • tester
    • One way descriptor isn’t good for general outlet detection
      • Will test the planar detector
  • Spin images
    • Generated a scene from a model and added noise
      • Tested spin image on this for geometric consistency and grouping of correspondences make sense
        • Artificial test worked fine when testing reprojection
        • But when used on real Ikea data (4 objects), but didn’t work as well
          • If radius is too large, clutter hurts, if too small, then not discriminative
          • But theoretically, occlusion shouldn’t be such a problem
          • Anatoly is working on 2 different paths: Why occlusion is hurting, and problem of small support.
    • Have done most of Johnson’s pipeline except the mesh verification
  • Planar detector
    • LePetit’s fern point classifier is now in
      • Multiplying probabilities give 90% accuracy, averaging gives much less 60%
    • C-descriptor is based on this, but not implemented yet
    • One way descriptor had poor performance
    • Victor will test on latest data
    • Will add c-descriptor
  • User contribution … designer is on vacation
  • Beta Release
    • Train cascade on more general object detection
    • Bundle adjustment, gave seminar today.
    • Documentation, started
    • Friday next week
  • Release
    • No major functionality to add
    • Python interface integrated
    • Add more documentation
    • Add more tests
    • Additional 3 weeks

Reports:

Vadim

  1. latest_tested_snapshot has been finally updated (for that OpenCV has been checked on 32-bit Vista (MSVC + GCC 4.3), 64-bit Ubuntu 9.04, 32-bit Ubuntu 8.04 and MacOSX 10.5. The snapshot includes working planar object detector based on the fern point classifier (it’s not the original c-descriptor, it will be added after beta), fixed HOG descriptor & detector, unified keypoint detectors (they all now output a vector of KeyPoint structure instances) and many bug fixes, new ones (more accurate cvLog(), corrected cvDrawContours etc.) and fixed earlier in trunk.
  2. added gzip support in OpenCV XML-YAML I/O functions. Now you can simply do:

cvSave(“bigmatrix.xml.gz”, mybigmatrix); and similarly mybigmatrix = (CvMat*)cvLoad(“bigmatrix.xml.gz”);

cvOpenFileStorage() and all other I/O functions calling it (including save() & load() methods in MLL classes) support this new feature. This was primarily done to reduce disk footprint for the new fern-based planar object detector, since it can easily take >200MB of disk space. When compressed, such file takes just ~20-30MB. Further reduction of memory/disk footprint by factor of ~4 is possible and will be done after beta release.

  1. fixed 3 bugs in highgui on Mac; now all the samples run fine.
  2. started updating the documentation with the new material on C++ interface; integrated the docs build into OpenCV CMake build system.
  3. received the final version of traincascade application + the detection code supporting LBP and Haar features. Integrating it now.

Victor

  • Outlet detection: a multiscale version of outlet detection has been implemented. Multiscale support has been added to one way descriptor — features are extracted now from several levels of image pyramid and descriptors are built for all of the features. The best match from any level determines a feature class (power hole/ground hole/background). Hough transform also operates on several levels of pyramid, scaling the template and matching it to each of pyramid levels, selecting the best solution by a score that depends on the number of features matched and reprojection error. As a result instead of 17 misdetections on white2x2 dataset (where scale variation is high — more than 2x) we get only 3 misdetections out of 105 images by using just 2 levels of a pyramid.
  • The application for automatic generation of outlet template has been created. It requires user to manually label the holes using a simple GUI. The same app has the functionality of automatically testing the performance of an outlet detector by comparing the results with the labels. All forearm data has been labeled and starting tomorrow we will use this app for automatically generating test statistics.
  • Spin images: geometric consistency filtering and grouping functionality has been tested. Affine transformation from scene to model using RANSAC has been implemented. It selects one group of correspondences by using a score that depends on the number of correspondences and reprojection error. Grouping is proved to give reasonable results for a scene that is obtained from a model by adding random noise to each 3D point. However we failed so far to build reasonable group of correspondences for Ikea object. If spin image support is small, we are getting too many correspondences because of symmetry and ambiguity of a local shape descriptor. If we increase spin image support, scene and model spin images differ a lot because part of the scene is occluded. We will investigate the small spin image support case and try to obtain a consistent group of correspondences.

Action Items

Gary

  • Send Victor wiki page documentation examples
  • Talk to Patrick, Marius about Nister trees
  • Talk to Marius about FLANN, Chamfer

Vadim

  • Send note to Marius about FLANN

PREVIOUSLY

Gary

  • Ask Brian about storing large data (outlet specific data — image plus hole data and then PCA components (small) and descriptors (large 100M uncompressed)

Vadim

  • Send reminder email to web designer about user contribution site

Victor

  • Add one way descriptor.

2009-08-04

Agenda

  • Outlet report
  • One way descriptor
  • User contribution
  • Daisy code is available, can try it
  • Bundle adjustment
  • Spin images
  • Documentation
  • Merging python in
  • {OK} FLANN, (./) Chamfer, ? Nister
  • 3D feature report
  • Post release
    • Static builds
    • 3D Capture
    • Alex’s feature engine

OpenCV Summer Release:

  • Dates:
    • Feature freeze July 31
    • Beta Release Aug 10.
    • Release August 31, 2009
      *
    • {OK} Done; (./) In progress; :\ Um, maybe; {X} Left as an exercise for the user to implement

PLANS:

  • General Structure:
    • :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
  • New Features
    • {OK} C++ interface
      • {OK} cxcore
      • {OK} cv
      • {OK} HighGUI
      • (./) I/O for sructures, histograms
    • Better python interface
      • {OK} In progress thanks to (James Bowman)
      • (./) Move into OpenCV and deprecate SWIG only version Post Beta
      • (./) Use this to expand unit test functions for OpenCV Post Beta
    • 2D features
      • {OK} HOG
        • (./) Chamfer matching with orientation In progress thanks to Marius Muja
      • (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
        • {OK} LePetit keypoint detector YAPE (Yet Another Point Extractor). We have found this to be the most stable detector.
        • (./) C-descriptor … (Vadim Michael — on site. Difficulties reproducing results)
        • {OK} Fern based planar patch detector from LePetit’s group
        • (./) LePetit one way descriptor (Victor) (by Aug 7)
      • {OK} MSER
      • {OK} Star
      • {OK} Self-similar detector
    • 3D features
      • (./) Bundle Adjustment (probable contribution by Phase Space … Talk to Sergey)
      • (./) normals in grid (Gary)
      • (./) normal based 3D patch (Gary)
      • {OK} SPIN
    • Optical flow
      • {OK} Dense (Farnerback)
    • MLL
      • {OK} Extreme trees
      • {OK} Reduce memory footprint
      • {OK} More convenient training via a data class
      • (./) Nister trees (Patrick) Post Beta
      • (./) FLANN or appropriate parts (Marius) Post Beta
      • {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)
  • Misc:
    • {OK} adaptive skin tracking
  • Optimizations
    • (./) Various functions, ongoing …
  • Documentation
    • {OK} Move to one Latex source
      • {OK} Use to generate HTML, PDF
    • {OK} Add search function
    • {OK} Add user feedback/documentation bug system
    • {OK} Proof read
    • {OK} Put up
      • {OK} cxcore
      • {OK} cv
      • {OK} MLL
      • {OK} Aux
      • {OK} HighGUI
    • (./) Add content
    • (./) Create C++ documentation
    • {OK} Create Python documenation
  • High GUI
    • {OK} In memory .jpg, .png
    • {OK} move to C++ interface
  • Testing functions
    • (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
  • (./) Define user contribution process:

Put off to Future

  • 2D features
    • Scale Invariant Feature Transform with Irregular Orientation Histogram Binning_, Yan Cui, Nils Hasler, Thorsten Thorm ?hlen, Hans-Peter Seidel (Feature_Point_Descriptors_SIFT_ImprovementCuiHasThoSei09igSIFT.pdf)
      • Alpha channel, alternatives to contours RLE
      • New alpha transparency (per pixel) added
      • Introduce functions with advance ways of blending
      • Mask channels
  • 3D features
    • Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
    • PFH, FPFH from Radu,
    • 3D Delaunay,
    • 3D convex hull,
    • mesh generation,
    • mesh simplification/subsampling
    • Extend to using 2D XYZA grid as well as oct/kd trees
  • 3D capture
    • Probably by using silhouettes and then texture for cavities (?)
      • Possibly with textures or phased light
    • VSLAM
    • Image stitching
  • Optical flow
    • Variational optical flow
  • MLL
    • Tiechman’s
      • Kernel based gentle boost
      • Feature selection engine
    • Automatic parameter tuning
      • Cross-validation, ROC curve generator
  • High GUI
    • move to using QT as the HighGUI interface look and feel
  • Misc:
    • Focus detector,
  • Lighting/Specularity
    • Look into:
      • Using Specularities for Recognition, Margarita Osadchy, David Jacobs, Ravi Ramamoorthi (stored on my disk as specularities.pdf)
      • Retrieving Multiple Light Sources in the Presence of Specular Reflections and Texture_, Pascal Lagger and Pascal Fua (Specular_multiple_sourcesFua08a-1.pdf)
      • 3D Pose Refinement from Reflections_, Pascal Lagger, Mathieu Salzmann, Vincent Lepetit, Pascal Fua (Specular_laggercvpr08.pdf)
      • Detecting Specular Surfaces on Natural Images, Andrey DelPozo, Silvio Savarese
  • Histograms
    • Insert value function with interpolation
    • Automatic bin size adjustment for recognition performance using a test set
  • OpenCV Ideas 2009

Minutes

One Way Descriptor

  • Extended to use keypoint class
    • Had to ripple through this change
  • 3 classes — writing example

Documentation

  • Need list of attributes for each function
  • Can this be done with semi-automatic script such as James

User contribution

  • Nothing yet on website

Daisy

  • Permission for BSD license

Bundle adjustment

  • Haven’t talked to Sergey
  • Supposedly finished
    • He will be online from tomorrow
    • False correspondences may or may not be handled
  • Use for STF on outlets from a robot

Chamfer, Nistor

  • Postpone until after Beta

Beta Release

  • Still need to integrate cascade training (today)
  • One way descriptor
  • End of this week

Final Release

  • Documentation
  • Python integration
  • Test functions
  • Bundle adjustment
  • + 3 weeks

Histograms

  • Read Write
    • C structures CvSave CvLoad
    • New data structures, no I/O yet, after Beta
  • Input data
  • Better performance by interpolation between 2 neighbor bins linear interpolation

Reports:

Vadim

  1. added C++ video capturing interface to highgui. Now the whole highgui is covered by the C++ API.
  2. Maria and I integrated the latest object detection code to OpenCV, it is in SVN together with sample face detection cascade built on LBP features. During integration the interface has been simplified a lot. The training application will be committed today. The new code performance is on par with the prev. C implementation, but OpenMP threading has not been added yet.
  3. wrote 8-page introduction section for the new chapter “C++ Interface” of opencv.pdf. Added description of Mat, Mat_, Rect, Point, Vector, Vec, MatND and SparseMat classes. The rest of cxcore reference is in progress.
  4. made the newly added flann bindings by Marius compile on Windows and Mac.

Gary

  • 3D surface plug is finished was not robust to stereo noise for similar shaped objects.
    • As bag of features it gets 87% on 3 confusable objects
  • I am 70% done trying instead to use extend PFH to whole objects

Victor

  1. Outlet detection: the most of the week has been spent to restructure the one way descriptor code in order to submit it to opencv. It took much more time than expected (4 days opposed to one). As a result we have three objects. CvOneWayDescriptor encapsulates functionality for individual descriptor. CvOneWayDescriptorBase is intended for storing a collection of descriptors, efficiently building them from precomputed PCA descriptors and finding the closest one. CvOneWayDescriptorObject encapsulates a collection of descriptors that have support for visual words. Multiscale support has been reimplemented for more elegant code. Now the code is ready for submission, every function is documented. The sample of usage is on the way.
  2. 3D descriptors: the dependence of the transformation from model to scene on the group of spin image correspondences has been implemented. It was found that the quality of the transformation does not correlate with the group size. The transformation that has lowest reprojection error is calculated on a group with 5 correspondences although there are groups with >100 correspondences. We came to the conclusion that we need interactive visualization in order to understand the matching process on Ikea dataset. Anatoly has now compiled ROS and rviz and is working to get spin image matching visualization there.

Other: Alexey is on vacations June 31-August 3rd.

Action Items

Gary

  • Talk to James about wiki additions to the manual.
  • Ask James about automating list of parameters
  • Ask Kurt about status of bundle adjustment
  • Talk to about C-descriptor

Vadim

  • Write Sergey a message about Bundle
    • Uses generated datasets
    • Ask about handling false correspondences
  • Update opencv monthly http://opencv.willowgarage.com/wiki/OpenCV%20Monthly

Victor

PREVIOUSLY

Gary

  • ? Send Victor wiki page documentation examples
  • Talk to Patrick, Marius about Nister trees
  • {OK} Talk to Marius about FLANN, Chamfer
  • Provide histogram specs

Vadim

  • {OK} Send note to Marius about FLANN

2009-08-25

Agenda

  • Floating point Floodfill (already in)
  • Progress reports
  • Python
  • Outlet

OpenCV Summer Release:

  • Dates:
    • Feature freeze July 31
    • Beta Release Aug 10.
    • Release August 31, 2009
  • Keys:
    • {OK} Done; (./) In progress; :\ Um, maybe; {X} Left as an exercise for the user to implement

PLANS:

  • General Structure:
    • :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
  • New Features
    • {OK} C++ interface
      • {OK} cxcore
      • {OK} cv
      • {OK} HighGUI
      • (./) I/O for sructures, histograms
    • Better python interface
      • {OK} In progress thanks to (James Bowman)
      • (./) Move into OpenCV and deprecate SWIG only version Post Beta
      • (./) Use this to expand unit test functions for OpenCV Post Beta
    • 2D features
      • {OK} HOG
        • (./) Chamfer matching with orientation In progress thanks to Marius Muja
      • (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
        • {OK} LePetit keypoint detector YAPE (Yet Another Point Extractor). We have found this to be the most stable detector.
        • (./) C-descriptor … (Vadim Michael — on site. Difficulties reproducing results)
        • {OK} Fern based planar patch detector from LePetit’s group
        • (./) LePetit one way descriptor (Victor) (by Aug 7)
      • {OK} MSER
      • {OK} Star
      • {OK} Self-similar detector
    • 3D features
      • (./) Bundle Adjustment (probable contribution by Phase Space … Talk to Sergey)
      • (./) normals in grid (Gary)
      • (./) normal based 3D patch (Gary)
      • {OK} SPIN
    • Optical flow
      • {OK} Dense (Farnerback)
    • MLL
      • {OK} Extreme trees
      • {OK} Reduce memory footprint
      • {OK} More convenient training via a data class
      • (./) Nister trees (Patrick) Post Beta
      • (./) FLANN or appropriate parts (Marius) Post Beta
      • {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)
  • Misc:
    • {OK} adaptive skin tracking
  • Optimizations
    • (./) Various functions, ongoing …
  • Documentation
    • {OK} Move to one Latex source
      • {OK} Use to generate HTML, PDF
    • {OK} Add search function
    • {OK} Add user feedback/documentation bug system
    • {OK} Proof read
    • {OK} Put up
      • {OK} cxcore
      • {OK} cv
      • {OK} MLL
      • {OK} Aux
      • {OK} HighGUI
    • (./) Add content
    • (./) Create C++ documentation
    • {OK} Create Python documenation
  • High GUI
    • {OK} In memory .jpg, .png
    • {OK} move to C++ interface
  • Testing functions
    • (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
  • (./) Define user contribution process:

Put off to Future

  • 2D features
    • Scale Invariant Feature Transform with Irregular Orientation Histogram Binning_, Yan Cui, Nils Hasler, Thorsten Thorm ?hlen, Hans-Peter Seidel (Feature_Point_Descriptors_SIFT_ImprovementCuiHasThoSei09igSIFT.pdf)
      • Alpha channel, alternatives to contours RLE
      • New alpha transparency (per pixel) added
      • Introduce functions with advance ways of blending
      • Mask channels
  • 3D features
    • Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
    • PFH, FPFH from Radu,
    • 3D Delaunay,
    • 3D convex hull,
    • mesh generation,
    • mesh simplification/subsampling
    • Extend to using 2D XYZA grid as well as oct/kd trees
  • 3D capture
    • Probably by using silhouettes and then texture for cavities (?)
      • Possibly with textures or phased light
    • VSLAM
    • Image stitching
  • Optical flow
    • Variational optical flow
  • MLL
    • Tiechman’s
      • Kernel based gentle boost
      • Feature selection engine
    • Automatic parameter tuning
      • Cross-validation, ROC curve generator
  • High GUI
    • move to using QT as the HighGUI interface look and feel
  • Misc:
    • Focus detector,
  • Lighting/Specularity
    • Look into:
      • Using Specularities for Recognition, Margarita Osadchy, David Jacobs, Ravi Ramamoorthi (stored on my disk as specularities.pdf)
      • Retrieving Multiple Light Sources in the Presence of Specular Reflections and Texture_, Pascal Lagger and Pascal Fua (Specular_multiple_sourcesFua08a-1.pdf)
      • 3D Pose Refinement from Reflections_, Pascal Lagger, Mathieu Salzmann, Vincent Lepetit, Pascal Fua (Specular_laggercvpr08.pdf)
      • Detecting Specular Surfaces on Natural Images, Andrey DelPozo, Silvio Savarese
  • Histograms
    • Insert value function with interpolation
    • Automatic bin size adjustment for recognition performance using a test set
  • OpenCV Ideas 2009

Minutes

Schedule:

  • Beta release on August 31st
  • Gold release on September 21st

For Beta

  • Integrate bundle adjustment
  • Finish documentation C++
  • Document new CvAux functions

For Gold

  • Improve documentation
  • Integrate new Python interface
    • James must look at the new C++ and how hard wrapppers would be for that
      • Can he redo what he did for the new interface
    • Add machine learning library
  • Expand test code

Need Dedicated server

  • Quad core server
  • Allocate Anatoly
    • for different platform
    • for ROS

Spin Images

  • Work well on synthetic data
  • they are parameter sensitive for noisy data
  • WPFH may work better (Gary working on)
    • Focus on

Vadim

  • Most time spend on C++ documentation
    • cxcore done
    • New cv module in process
  • Fixed some bugs
    • SVM old bug fixed
    • EM bug fixed Ira Shevtsova
    • Anatoly reported reference cout bug … will be fixed involves thread safety for reference count
  • The threaded version of the new object detection methods was improved and committed to SVN (Maria Dimashova).
    • We also developed the new internal interfaces for threading the decision tree training algorithm (optimal split selection), which is also the core of boosting and random trees training. Maria is now working on implementing it.
  • Bundle adjustment code is in from Sergey
    • Sergey implemented test code for opencv test system

Victor:

Outlet detection: several problems with the one way descriptor matching have been corrected. A strong shadow cast on an outlet caused incorrect classification of outlet holes (ground/power mismatch). This was fixed by adding a variation in scale for matching. Also, patch resolution has been increased 2 times, from 24×24 up to 48×48. The resulting method works significantly better on orange2x1, solving the most of the shadow problems. The testing framework has been fully automated allowing to achieve statistics for all of the datasets. Below are the results for the older and newer methods.

Improved One-way Descriptor

fake_wall_2x1white 97/114
fake_wall_2x2orange 33/35
orange2x1 97/103
specular 25/72
white2x1hall 94/103
white2x2 100/107

Old One-way Descriptor

fake_wall_2x1white 96/114
fake_wall_2x2orange 34/35
orange2x1 89/103
specular 26/72
white2x1hall 97/103
white2x2 97/107

3D feature matching: VRML rendering for debugging matching results has been implemented. Several methods for limiting the class of the resulting affine transformations (model → scene) have been implemented. We have failed to get Johnson algorithm for spin image matching to work in generic setting. The main problems are shape ambiguity (model objects have axial symmetry) and occlusion (shape on the scene is not full, absence of the points influences spin images). Probably the usage of prior knowledge (planar geometry from the table, axial symmetry) as well as image feature matching will help to solve the practical problem.

= To Do =

Victor

  • Send in flow chart of outlet matching
    • As done on milestone
    • As done now

Gary

  • Ask about outlet plugging in plans
  • Ask clean workstation ROS
    • Jaunty

2009-09-01

Agenda

  • Schedule
  • Test system
  • 3D capture
  • Outlets

Schedule:

  • Beta release on August 31st Sept 4th due to Bundle and C++
  • Gold release on September 21st

For Beta

  • Bundle adjustment (failed)
  • Finish documentation C++
  • Failure on mingw
  • {OK} Simplify the OpenCV wiki homepage.

For Gold

  • Improve documentation
  • Integrate new Python interface
    • James must look at the new C++ and how hard wrapppers would be for that
      • Can he redo what he did for the new interface
    • Add machine learning library
  • Expand test code

OpenCV Summer Release:

  • Dates:
    • Feature freeze July 31
    • Beta Release Aug 10.
    • Release August 31, 2009
  • Keys:
    • {OK} Done; (./) In progress; :\ Um, maybe; {X} Left as an exercise for the user to implement

PLANS:

  • General Structure:
    • :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
  • New Features
    • {OK} C++ interface
      • {OK} cxcore
      • {OK} cv
      • {OK} HighGUI
      • (./) I/O for sructures, histograms
    • Better python interface
      • {OK} In progress thanks to (James Bowman)
      • (./) Move into OpenCV and deprecate SWIG only version Post Beta
      • (./) Use this to expand unit test functions for OpenCV Post Beta
    • 2D features
      • {OK} HOG
        • (./) Chamfer matching with orientation In progress thanks to Marius Muja
      • (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
        • {OK} LePetit keypoint detector YAPE (Yet Another Point Extractor). We have found this to be the most stable detector.
        • (./) C-descriptor … (Vadim Michael — on site. Difficulties reproducing results)
        • {OK} Fern based planar patch detector from LePetit’s group
        • (./) LePetit one way descriptor (Victor) (by Aug 7)
      • {OK} MSER
      • {OK} Star
      • {OK} Self-similar detector
    • 3D features
      • (./) Bundle Adjustment (probable contribution by Phase Space … Talk to Sergey)
      • (./) normals in grid (Gary)
      • (./) normal based 3D patch (Gary)
      • {OK} SPIN
    • Optical flow
      • {OK} Dense (Farnerback)
    • MLL
      • {OK} Extreme trees
      • {OK} Reduce memory footprint
      • {OK} More convenient training via a data class
      • (./) Nister trees (Patrick) Post Beta
      • (./) FLANN or appropriate parts (Marius) Post Beta
      • {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)
  • Misc:
    • {OK} adaptive skin tracking
  • Optimizations
    • (./) Various functions, ongoing …
  • Documentation
    • {OK} Move to one Latex source
      • {OK} Use to generate HTML, PDF
    • {OK} Add search function
    • {OK} Add user feedback/documentation bug system
    • (./) Proof read — Needs complet re-do
    • {OK} Put up
      • {OK} cxcore
      • {OK} cv
      • {OK} MLL
      • {OK} Aux
      • {OK} HighGUI
    • (./) ’C++ Interface
      • {OK} C++ cxcore
      • (./) cv (90%)
      • (./) Aux
      • (./) HighGUI
    • (./) Add content
    • (./) Create C++ documentation
    • {OK} Create Python documenation
  • High GUI
    • {OK} In memory .jpg, .png
    • {OK} move to C++ interface
  • Testing functions
    • (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
  • (./) Define user contribution process:

Put off to Future

  • 2D features
    • Scale Invariant Feature Transform with Irregular Orientation Histogram Binning_, Yan Cui, Nils Hasler, Thorsten Thorm ?hlen, Hans-Peter Seidel (Feature_Point_Descriptors_SIFT_ImprovementCuiHasThoSei09igSIFT.pdf)
      • Alpha channel, alternatives to contours RLE
      • New alpha transparency (per pixel) added
      • Introduce functions with advance ways of blending
      • Mask channels
  • 3D features
    • Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
    • PFH, FPFH from Radu,
    • 3D Delaunay,
    • 3D convex hull,
    • mesh generation,
    • mesh simplification/subsampling
    • Extend to using 2D XYZA grid as well as oct/kd trees
  • 3D capture
    • Probably by using silhouettes and then texture for cavities (?)
      • Possibly with textures or phased light
    • VSLAM
    • Image stitching
  • Optical flow
    • Variational optical flow
  • MLL
    • Tiechman’s
      • Kernel based gentle boost
      • Feature selection engine
    • Automatic parameter tuning
      • Cross-validation, ROC curve generator
  • High GUI
    • move to using QT as the HighGUI interface look and feel
  • Misc:
    • Focus detector,
  • Lighting/Specularity
    • Look into:
      • Using Specularities for Recognition, Margarita Osadchy, David Jacobs, Ravi Ramamoorthi (stored on my disk as specularities.pdf)
      • Retrieving Multiple Light Sources in the Presence of Specular Reflections and Texture_, Pascal Lagger and Pascal Fua (Specular_multiple_sourcesFua08a-1.pdf)
      • 3D Pose Refinement from Reflections_, Pascal Lagger, Mathieu Salzmann, Vincent Lepetit, Pascal Fua (Specular_laggercvpr08.pdf)
      • Detecting Specular Surfaces on Natural Images, Andrey DelPozo, Silvio Savarese
  • Histograms
    • Insert value function with interpolation
    • Automatic bin size adjustment for recognition performance using a test set
  • OpenCV Ideas 2009

Minutes

Reference counting and Python

  • Does it conflict?
  • Use numerical numpy array before calling OpenCV, OpenCV can point a header at it. This shuts off OpenCV reference counting.

Talked with NVidia about OpenCV support

  • He will discuss with NVidia management
  • There is also OpenCL Which abstracts “CUDA” code

New workstation

  • MacPro with 8 core, each at 2.8GHz, with lots of cache and memory
  • Can test multi-threaded scaling behaviour beyond dual core
  • Next year, Larrabee will be out (at least 16 cores, maybe 32 core version)

Currently, team members are on:

  • Vadim
    • documentation et al
  • Victor
    • outlet
  • Alexy
    • outlet also, pose estimation, generalized Hough
  • Maria
    • parallel MLL
  • Anatoly
    • Spin images might switch to developing test system (unless we hire Andrew for 2 months)

Test system with ROS

  • 2-4 weeks
    • Test with ROS
    • Making automatic tests
    • Documentation builds
    • Automate trunk→latest tested snapshot promotion

Changes in user group and main page

  • We (James, Gary) simplified the main wiki page to make the main things (docs, download and user group) easier to find
  • I made the messages on the Yahoo groups readable by anyone so that Google, Bing can index them.

One way descriptor

  • Does affine distortion and can use the match to estimate pose

Spin images aren’t very discriminative on stereo scans of glasses since the legs don’t show up

3Dsom

Vadim

The following has been done last week:

  1. The operations on Mat::refcount made thread-safe; now they use atomic increment & decrement operations. That eliminated some crashes we get in some recently added multi-threaded code (like spin images and parallel cascade training).
  2. Bundle adjustment has been integrated and put to SVN. Tests compile too, but they fail currently. Need to study that.
  3. C++ Documentation for cv is nearly (~90%) complete. Documentation for cvaux and highgui has not been updated yet.
  4. Maria implemented added OpenMP-parallelized code to the best split selection methods in MLL’s CvDTree (decision tree). So now we have parallel decision tree construction, boosting & random trees training, and also the parallel traincascade application.
    • On the traincascade we get especially good results with OpenMP, about ~1.9 speedup on dual core machines.
  5. When OpenCV is built with Mingw (gcc for windows), the current ML tests crash or freeze. It is likely because of recently added threading.
    • Maria is now studying the problem.

Victor

  • Outlet detection: the detector based on one way descriptor was improved by tuning several parameters, the latest results are in the table below. Lower detection rate on orange2x1 is due to one way descriptor instability caused by strong shades, one way descriptor does not match correctly the points in specular because of extreme lighting.

Using Generalized Hough Voting


cracked (correct/total): 99/100 99.0%
fake_wall_2x1white (correct/total): 109/111 98.2%
fake_wall_2x2orange (correct/total): 34/34 100.0%
orange2x1 (correct/total): 98/103 95.2%
specular (correct/total): 24/56 42.9% (extreme lighting)
white2x1hall (correct/total): 101/103 98.1%
white2x2 (correct/total): 102/104 98.1%
  • Pose description obtained from one way descriptor was used to detect the outlet. A template was transformed according to a pose from each of the detected keypoints and template keypoints were matched with the keypoints from the test image. The pose with the largest match count was taken as the final answer result. The results are only slightly worse than hough transform:

Using the pose estimation from the one way descriptor


cracked (correct/total): 94/100
fake_wall_2x1white (correct/total): 110/111
fake_wall_2x2orange (correct/total): 34/34
orange2x1 (correct/total): 96/103
specular (correct/total): 22/56
white2x1hall (correct/total): 102/103
white2x2 (correct/total): 91/104
  • Speed of one way descriptor
    • 0.5-2 seconds total detection time for 640×480 Can possibly improved, might be in memory allocation.
    • Hough was sped up to ~0.1 secs so it is no longer the bottleneck
  • Foils for the orange outlet detector method and one way descriptor method have been prepared.
  • Spin image matching: several bugs in spin image filtering have been found. We have proved that in spite of the parameter tuning spin images do not produce matches that can be used to find a correct transform from a model to a scene. The reason is that the whole scene surface where spin images can be calculated is the same locally resulting in similar shape descriptors that are biased by unstable normals and noise in point cloud.

Action Items

Gary

  • Talk with James about use of numpy arrays and having OpenCV wrap them
  • Get 3D data to Victor

Vadim * Have Sergey look at bundle adjustment

  • Get out Beta

From Last time

Victor

  • Send in flow chart of outlet matching
    • As done on milestone
    • As done now

Gary

  • Ask about outlet plugging in plans
  • - Ask clean workstation ROS -
    • -Jaunty -

2009-09-08

Agenda

  • Schedule
  • Bundle adjustment
  • Outlets

Schedule:

  • Beta release on August 31st Sept 4th due to Bundle and C++ Sept 9th
  • Gold release on September 21st

For Beta

  1. build the installation package on Windows – since we migrated from InnoSetup to NSIS + CMake, that requires some extra work on the CMake scripts
    • We’ve moved to cpack (failed last night due to the python interface) # update the ChangeLog (now 50% done), the INSTALL document and doc/index.html.
  2. repair the autotools + configure script (or put the note).
  3. Tuesday Morning, Sept. 9th, 2009 Pacific Time

For Gold

  • Improve documentation
  • Integrate new Python interface
    • James must look at the new C++ and how hard wrapppers would be for that
      • Can he redo what he did for the new interface
    • Add machine learning library
  • Expand test code

OpenCV Summer Release:

  • Dates:
    • Feature freeze July 31
    • Beta Release Aug 10.
    • Release August 31, 2009
  • Keys:
    • {OK} Done; (./) In progress; :\ Um, maybe; {X} Left as an exercise for the user to implement

PLANS:

  • General Structure:
    • :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
  • New Features
    • {OK} C++ interface
      • {OK} cxcore
      • {OK} cv
      • {OK} HighGUI
      • (./) I/O for sructures, histograms
    • Better python interface
      • {OK} In progress thanks to (James Bowman)
      • (./) Move into OpenCV and deprecate SWIG only version Post Beta
      • (./) Use this to expand unit test functions for OpenCV Post Beta
    • 2D features
      • {OK} HOG
        • (./) Chamfer matching with orientation In progress thanks to Marius Muja
      • (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
        • {OK} LePetit keypoint detector YAPE (Yet Another Point Extractor). We have found this to be the most stable detector.
        • (./) C-descriptor … (Vadim Michael — on site. Difficulties reproducing results)
        • {OK} Fern based planar patch detector from LePetit’s group
        • (./) LePetit one way descriptor (Victor) (by Aug 7)
      • {OK} MSER
      • {OK} Star
      • {OK} Self-similar detector
    • 3D features
      • (./) Bundle Adjustment (probable contribution by Phase Space … Talk to Sergey)
      • (./) normals in grid (Gary)
      • (./) normal based 3D patch (Gary)
      • {OK} SPIN
    • Optical flow
      • {OK} Dense (Farnerback)
    • MLL
      • {OK} Extreme trees
      • {OK} Reduce memory footprint
      • {OK} More convenient training via a data class
      • (./) Nister trees (Patrick) Post Beta
      • (./) FLANN or appropriate parts (Marius) Post Beta
      • {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)
  • Misc:
    • {OK} adaptive skin tracking
  • Optimizations
    • (./) Various functions, ongoing …
  • Documentation
    • {OK} Move to one Latex source
      • {OK} Use to generate HTML, PDF
    • {OK} Add search function
    • {OK} Add user feedback/documentation bug system
    • (./) Proof read — Needs complet re-do
    • {OK} Put up
      • {OK} cxcore
      • {OK} cv
      • {OK} MLL
      • {OK} Aux
      • {OK} HighGUI
    • (./) ’C++ Interface
      • {OK} C++ cxcore
      • (./) cv (90%)
      • (./) Aux
      • (./) HighGUI
    • (./) Add content
    • (./) Create C++ documentation
    • {OK} Create Python documenation
  • High GUI
    • {OK} In memory .jpg, .png
    • {OK} move to C++ interface
  • Testing functions
    • (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
  • (./) Define user contribution process:

Put off to Future

  • 2D features
    • Scale Invariant Feature Transform with Irregular Orientation Histogram Binning_, Yan Cui, Nils Hasler, Thorsten Thorm ?hlen, Hans-Peter Seidel (Feature_Point_Descriptors_SIFT_ImprovementCuiHasThoSei09igSIFT.pdf)
      • Alpha channel, alternatives to contours RLE
      • New alpha transparency (per pixel) added
      • Introduce functions with advance ways of blending
      • Mask channels
  • 3D features
    • Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
    • PFH, FPFH from Radu,
    • 3D Delaunay,
    • 3D convex hull,
    • mesh generation,
    • mesh simplification/subsampling
    • Extend to using 2D XYZA grid as well as oct/kd trees
  • 3D capture
    • Probably by using silhouettes and then texture for cavities (?)
      • Possibly with textures or phased light
    • VSLAM
    • Image stitching
  • Optical flow
    • Variational optical flow
  • MLL
    • Tiechman’s
      • Kernel based gentle boost
      • Feature selection engine
    • Automatic parameter tuning
      • Cross-validation, ROC curve generator
  • High GUI
    • move to using QT as the HighGUI interface look and feel
  • Misc:
    • Focus detector,
  • Lighting/Specularity
    • Look into:
      • Using Specularities for Recognition, Margarita Osadchy, David Jacobs, Ravi Ramamoorthi (stored on my disk as specularities.pdf)
      • Retrieving Multiple Light Sources in the Presence of Specular Reflections and Texture_, Pascal Lagger and Pascal Fua (Specular_multiple_sourcesFua08a-1.pdf)
      • 3D Pose Refinement from Reflections_, Pascal Lagger, Mathieu Salzmann, Vincent Lepetit, Pascal Fua (Specular_laggercvpr08.pdf)
      • Detecting Specular Surfaces on Natural Images, Andrey DelPozo, Silvio Savarese
  • Histograms
    • Insert value function with interpolation
    • Automatic bin size adjustment for recognition performance using a test set
  • OpenCV Ideas 2009

Minutes

The current activities (TBD before beta):

  1. build the installation package on Windows – since we migrated from InnoSetup to NSIS + CMake, that requires some extra work on the CMake scripts
    • We’ve moved to cpack (failed last night due to the python interface)
  2. update the ChangeLog (now 50% done), the INSTALL document and doc/index.html.
  3. repair the autotools + configure script (or put the note).
  4. Tuesday Morning, Sept. 9th, 2009 Pacific Time

Python

  • Currently requires manual work for each function, but as API stabilizes, this shouldn’t matter
  • Need to get a defined way to use the new C++ API in OpenCV

New machine

  • Got it, set it up
  • Anatoly is now working on the new build system.

Outlets

  • Goal: 2×1 white on white
    • Either have template or send image
  • Slow accurate code is 1.5 sec/frame, fast is 0.5 seconds per frame.
    • Use better detector first
  • No sub-pixel estimation
    • Detects small, dark connected components. Very lighting dependent on direction of the light.
    • Width of the slot hole 3-5 pixels
    • But, once we have the holes and the template, we do a homography estimation
  • Input is an image of an outlet + outlet template
    • Output gives you an array of structures containing coordinates of the holes in 2 and 3D plus hole contrast etc
  • Chessboard
    • Victor sent plug.cpp

Outlets

Victor

  • Outlet detection: one way descriptor has been optimized. Algorithmic optimization and parameter tuning (reduction of low resolution pca from 100 to 20 components as well as reduction of the pose number from 500 to 100) resulted in >3x speedup (0.5s down from 1.5s per image). L1 norm did not give any speedup relative to L2. Accuracy dropped slightly, see the tables below. Now we are looking at alternative techniques that are faster than one way descriptor (ferns and binary matching — S.Taylor et al “Robust feature matching in 2.3 microseconds”, feature descriptors workshop @ CVPR’09). The outlet template generator has been integrated into a single application that can be used to mark the holes and calculate outlet physical dimensions.
  • Plug detection: short experiment with a small 4×3 chessboard showed that a simple findchessboard + findextrinsiccameraparams gives a stable 3D coordinates of the chessboard relative to the camera (wide-angle iSight from my MacBook Pro) when the chessboard is 10-20 cm away.

Outlet detection accuracy before optimisation (~1.5 sec/frame):


cracked (correct/total): 94/100
fake_wall_2x1white (correct/total): 110/111
fake_wall_2x2orange (correct/total): 34/34
orange2x1 (correct/total): 96/103
specular (correct/total): 22/56
white2x1hall (correct/total): 102/103
white2x2 (correct/total): 91/104

Outlet detection accuracy after optimisation (~0.5 sec/frame):


cracked (correct/total): 91/100
fake_wall_2x1white (correct/total): 110/111
fake_wall_2x2orange (correct/total): 33/34
orange2x1 (correct/total): 96/103
specular (correct/total): 23/56
white2x1hall (correct/total): 102/103
white2x2 (correct/total): 89/104

Outlet detection accuracy when GHT is used for outlet pose estimation instead of one way descriptor (GHT takes an additional 0.5-1s per frame)


cracked (correct/total): 99/100 | Average time: 1.386944
fake_wall_2x1white (correct/total): 109/111 | Average time: 1.005288
fake_wall_2x2orange (correct/total): 33/34 | Average time: 1.938700
orange2x1 (correct/total): 97/103 | Average time: 0.723746
specular (correct/total): 25/56 | Average time: 0.886223
white2x1hall (correct/total): 102/103 | Average time: 0.554561
white2x2 (correct/total): 103/104 | Average time: 1.654778

Vadim:

  1. The whole documentation (cxcore, cv, highgui) is ready and is in SVN. cvaux and mll documentation updates to be done after beta release.
  2. A serious bug in filtering functions has been fixed; about a dozen of other smaller (like inconsistent APIs, duplicating functions …) have been fixed during the documentation updates. Another bug in the boosting save/load functions, reported by Rainer, has been fixed.
  3. The documentation has been moved from doc2 to doc; the old HTML reference has been removed.
  4. Ran the tests on Windows, Linux (64-bit) and MacOSX, ran the samples. Everything works well.

Action Items

Gary

  • Ask James about automating python (how automatic)
    • C++ for Python
  • Ask Patrick what he did with the chessboard and tip of plug
  • Get images to Victor

Vadim

  • Get out Beta
  • Ask Sergey to help make Bundle adjustment work (generate test data automatically)

Victor

  • Ask Sergey to help make Bundle adjustment work (generate test data automatically)

From Last time

Gary

  • {OK} Talk with James about use of numpy arrays and having OpenCV wrap them
  • {X} Get 3D data to Victor

Vadim * Have Sergey look at bundle adjustment

  • {X} Get out Beta

2009-09-15

Agenda

Release

  • What needs to be done for Gold release
    • Realistic Schedule
  • Bundle adjustment
  • Test system/new machine
  • Go through list below
  • Update website, announce to the group …

Outlets

  • Outlet progress on the bad images
  • Results over all orange images?

Schedule:

  • Beta 1.2.0 is released! Sept 14th
  • Gold release on September 29st, announce at ICCV 2009

For Gold 2.0

  1. Fix installation bugs
  2. Documentation
    • Fix formulas
    • Fix links
    • Improving references
  3. Anatoly and Maria are creating and running tests of OpenCV in various condigurations, OS et
  4. Bug tracker clean out

OpenCV Summer Release:

  • Dates:
    • Feature freeze July 31
    • Beta Released: Sept 14th.
    • Gold Release Sept 31, 2009
  • Keys:
    • {OK} Done; (./) In progress; :\ Um, maybe; {X} Left as an exercise for the user to implement

PLANS:

  • General Structure:
    • :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
  • New Features
    • {OK} C++ interface
      • {OK} cxcore
      • {OK} cv
      • {OK} HighGUI
      • {OK} I/O for sructures, histograms
    • Better python interface
      • {OK} In progress thanks to (James Bowman)
      • {OK} Move into OpenCV and deprecate SWIG only version → selected in CMake either or both
      • {OK} Use this to expand unit test functions for OpenCV
    • 2D features
      • {OK} HOG
        • {X} Chamfer matching with orientation In progress thanks to Marius Muja
      • (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
        • {OK} LePetit keypoint detector YAPE * (Yet Another Point Extractor). We have found this to be the most stable detector.
        • {X} C-descriptor … (Vadim Michael — on site. Difficulties reproducing results)
        • {OK} Fern based planar patch detector from LePetit’s group D
        • {OK} LePetit one way descriptor (Victor) D
      • {OK} MSER — list of contours … how to extract keypoint from this
      • {OK} Fast detector *
      • {OK} SURF * D
      • {OK} Star *
      • {OK} Self-similar detector
    • 3D features
      • {OK} Bundle Adjustment (by Phase Space / Sergey)
      • {X} normals in grid (Gary)
      • {X} normal based 3D patch (Gary)
      • {OK} SPIN
    • Optical flow
      • {OK} Dense (Farnerback)
    • MLL
      • {OK} Extreme trees
      • {OK} Reduce memory footprint
      • {OK} More convenient training via a data class
      • {X} Nister trees (Patrick) Post Beta
      • {OK} FLANN or appropriate parts (Marius)
      • {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)
  • Misc:
    • {OK} adaptive skin tracking
  • Documentation
    • {OK} Move to one Latex source
      • {OK} Use to generate HTML, PDF
    • {OK} Add search function
    • {OK} Add user feedback/documentation bug system
    • (./) Proof read — Needs complet re-do
    • {OK} Put up
      • {OK} cxcore
      • {OK} cv
      • {OK} MLL
      • {OK} Aux
      • {OK} HighGUI
    • (./) ’C++ Interface
      • {OK} C++ cxcore
      • {OK} cv
      • (./) Aux nothing for new functions
      • {OK} HighGUI
      • (./) Needs improvement on class descriptions
    • {OK} Create Python documenation
  • High GUI
    • {OK} In memory .jpg, .png
    • {OK} move to C++ interface
    • {X} visualizations, statistical data extensions
  • Testing functions
    • (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
    • {OK} Bought new server for automated tests and builds
      • (./) creating auto test system → Usable results 3 weeks
  • (./) Define user contribution process:

Put off to Future

  • 2D features
    • Scale Invariant Feature Transform with Irregular Orientation Histogram Binning_, Yan Cui, Nils Hasler, Thorsten Thorm ?hlen, Hans-Peter Seidel (Feature_Point_Descriptors_SIFT_ImprovementCuiHasThoSei09igSIFT.pdf)
      • Alpha channel, alternatives to contours RLE
      • New alpha transparency (per pixel) added
      • Introduce functions with advance ways of blending
      • Mask channels
  • 3D features
    • Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
    • PFH, FPFH from Radu,
    • 3D Delaunay,
    • 3D convex hull,
    • mesh generation,
    • mesh simplification/subsampling
    • Extend to using 2D XYZA grid as well as oct/kd trees
  • 3D capture
    • Probably by using silhouettes and then texture for cavities (?)
      • Possibly with textures or phased light
    • VSLAM
    • Image stitching
  • Optical flow
    • Variational optical flow
  • MLL
    • Tiechman’s
      • Kernel based gentle boost
      • Feature selection engine
    • Automatic parameter tuning
      • Cross-validation, ROC curve generator
  • High GUI
    • move to using QT as the HighGUI interface look and feel
  • Misc:
    • Focus detector,
  • Lighting/Specularity
    • Look into:
      • Using Specularities for Recognition, Margarita Osadchy, David Jacobs, Ravi Ramamoorthi (stored on my disk as specularities.pdf)
      • Retrieving Multiple Light Sources in the Presence of Specular Reflections and Texture_, Pascal Lagger and Pascal Fua (Specular_multiple_sourcesFua08a-1.pdf)
      • 3D Pose Refinement from Reflections_, Pascal Lagger, Mathieu Salzmann, Vincent Lepetit, Pascal Fua (Specular_laggercvpr08.pdf)
      • Detecting Specular Surfaces on Natural Images, Andrey DelPozo, Silvio Savarese
  • Histograms
    • Insert value function with interpolation
    • Automatic bin size adjustment for recognition performance using a test set
  • OpenCV Ideas 2009

Minutes

Documentation

  • Separate chapter for C++, not up yet
    • Not complete, functions are described
    • Classes are not described
  • Proposal
    • Do we have 3 manuals? C++, Python, C
    • By conditionals in the Latex source?
    • Update doc continuously
  • Bug in Platex doesn’t seem to work.
    • M4? But not on Windows
  • Talk about fix online

Bugs

  • > 100, need to clear out

C++ Python interface

  • Functions in C are still functions in C++, but inside namespace cv
    • Python interface for these can look the same
    • One important difference for new C++ interface, output is always Mat
      • Mat will be of same size as input of type specified
      • For python, people use PIL
        • All operators are methods of the image
        • Python people would be used to this
  • But there are quite a few real classes.
    • MLL is an example
  • How to make Python and C++ intefaces more alike
    • Python should wrap he C++
    • ptrs to Mat are not referenced counted in C++
    • Python keeps points to python and to C matrices
    • All new functionality is in C++
      • dense optical flow
      • HOG
      • One way descriptor
    • Vadim can take a C++ function, dense optical flow and see how this will work in Python

Versions

  • Beta is 1.2.0
  • Gold will be 2.0
  • Clear out old release

Bundle adjustment

  • Sergey is working on auto data generation for test
  • But the code should work
  • Documentation
    • Needs to be added
    • Needs sample function

Victor

  • Outlet detection: experiments on the new dataset showed sensitivity of one way descriptor to a change in intensity.
    • Matching of low brightness features from the new dataset and older bright template does not work. If a template is chosen from the new dataset, matching is much better, resulting in higher detection (below).
  • Automatic Template Selection is Possible: Experiments with detecting an outlet without knowing the correct template have been performed. A detector was run with each template, one by one, until an outlet is detected. This algorithm resulted in correct detections for the most of 2×1 outlets and bad detection on 2×2 outlets. The next step here is to run the detector with all the templates and choose the detection that has the higher likelihood (this is on the way).
    • This algorithm will allow us to automatically detect the correct template and then use it with no performance overhead.
  • Plug detection: chessboard detection does not work on the images from the new dataset — chessboard is too low resolution. However all black squares are detected with a hole detector (designed for outlet hole detection). So I am working now on a “low resolution chessboard detector”. The detection part is finished, ordering algorithm is on the way (standard ordering algorithm from cvFindChessboardCorners is hard to use due to differences in corners and black square centers).

New data with template taken from one of the images


fake_wall_2x1white_plug (correct/total): 34/38 | Average time: 0.964066
fake_wall_2x2orange_plug (correct/total): 20/21 | Average time: 3.460710

Vadim:

The following has been done last week:

  1. OpenCV 2.0 beta has been released.
    • The source package for Linux/Mac/*BSD and the binary package for Windows are available at https://sourceforge.net/projects/opencvlibrary/files/
    • The corresponding tag 2.0beta has been added to the SVN repository. # Two MacOSX-specific bugs have been fixed after Windows release and before the Linux one: altivec and STL conflict in cvSmooth source code and ffmpeg compilation (ported from ROS).

The current post-beta activities:

  1. I have read a few reports about running the precompiled OpenCV binaries on machines without Visual Studio installed, because of missing runtime libraries. While the users already posted a workaround for this problem, I’m now fixing this problem in the packaging script.
  2. The documentation needs some fixes (already found a few bugs in the formulae), there are missing descriptions for some new classes and functions, quite a few links do not work, and the bibliography is not properly formatted; working on that too.
  3. Will study the new Python bindings (cvpy) and figure out how to extend it to the C++ classes from MLL, cv and cvaux.

Action Items

Gary

  • Ask Marius about Chamfer
  • Ask about FLANN docs
    • There is sample code
  • Connect Victor and Jack

James

  • Work through some documentation examples for C++
  • Talk with Vadim about docs on line
  • Go through bugs, assign
  • Talk with Vadim about how to make Python and C++ interface more the same

Vadim

  • Talk to James about docs on line
  • Talk with James about how to make Python and C++ interface more the same

Victor

From Last time

Gary

  • -Ask James about automating python (how automatic) -
    • C++ for Python ’MOVE OFF OF C
  • Ask Patrick what he did with the chessboard and tip of plug
  • Get images to Victor

Vadim * Get out Beta

  • Ask Sergey to help make Bundle adjustment work (generate test data automatically)

Victor

  • Ask Sergey to help make Bundle adjustment work (generate test data automatically)

2009-09-22

Agenda

Release

  • Bundle Adjustment
  • Contributor acknowledgement
  • Schedule for Gold release?
  • Python etc
  • Code change statistics
  • IPP/MKL
  • 64 bit support

Outlets and Doorhandles

  • Results on new arm data

Schedule:

  • Beta 1.2.0 is released! Sept 14th
  • Gold release on September 29st, announce at ICCV 2009

For Gold 2.0

  1. Fix installation bugs
  2. Documentation
    • Fix formulas
    • Fix links
    • Improving references
  3. Anatoly and Maria are creating and running tests of OpenCV in various condigurations, OS et
  4. Bug tracker clean out

OpenCV Summer Release:

  • Dates:
    • {OK} Feature freeze July 31
    • {OK} Beta Released: Sept 14th.
    • Gold Release Sept 29, 2009
  • Keys:
    • {OK} Done; (./) In progress; :\ Um, maybe; {X} Left as an exercise for the user to implement

PLANS:

  • General Structure:
    • :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
  • New Features
    • {OK} C++ interface
      • {OK} cxcore
      • {OK} cv
      • {OK} HighGUI
      • {OK} I/O for sructures, histograms
    • Better python interface
      • {OK} In progress thanks to (James Bowman)
      • {OK} Move into OpenCV and deprecate SWIG only version → selected in CMake either or both
      • {OK} Use this to expand unit test functions for OpenCV
    • 2D features
      • {OK} HOG
        • {X} Chamfer matching with orientation In progress thanks to Marius Muja
      • (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
        • {OK} LePetit keypoint detector YAPE * (Yet Another Point Extractor). We have found this to be the most stable detector.
        • {X} C-descriptor … (Vadim Michael — on site. Difficulties reproducing results)
        • {OK} Fern based planar patch detector from LePetit’s group D
        • {OK} LePetit one way descriptor (Victor) D
      • {OK} MSER — list of contours … how to extract keypoint from this
      • {OK} Fast detector *
      • {OK} SURF * D
      • {OK} Star *
      • {OK} Self-similar detector
    • 3D features
      • {OK} Bundle Adjustment (by Phase Space / Sergey)
      • {X} normals in grid (Gary)
      • {X} normal based 3D patch (Gary)
      • {OK} SPIN
    • Optical flow
      • {OK} Dense (Farnerback)
    • MLL
      • {OK} Extreme trees
      • {OK} Reduce memory footprint
      • {OK} More convenient training via a data class
      • {X} Nister trees (Patrick) Post Beta
      • {OK} FLANN or appropriate parts (Marius)
      • {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)
  • Misc:
    • {OK} adaptive skin tracking
  • Documentation
    • {OK} Move to one Latex source
      • {OK} Use to generate HTML, PDF
    • {OK} Add search function
    • {OK} Add user feedback/documentation bug system
    • (./) Proof read — Needs complet re-do
    • {OK} Put up
      • {OK} cxcore
      • {OK} cv
      • {OK} MLL
      • {OK} Aux
      • {OK} HighGUI
    • (./) ’C++ Interface
      • {OK} C++ cxcore
      • {OK} cv
      • (./) Aux nothing for new functions
      • {OK} HighGUI
      • (./) Needs improvement on class descriptions
    • {OK} Create Python documenation
  • High GUI
    • {OK} In memory .jpg, .png
    • {OK} move to C++ interface
    • {X} visualizations, statistical data extensions
  • Testing functions
    • (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
    • {OK} Bought new server for automated tests and builds
      • (./) creating auto test system → Usable results 3 weeks
  • (./) Define user contribution process:

Put off to Future

  • 2D features
    • Scale Invariant Feature Transform with Irregular Orientation Histogram Binning_, Yan Cui, Nils Hasler, Thorsten Thorm ?hlen, Hans-Peter Seidel (Feature_Point_Descriptors_SIFT_ImprovementCuiHasThoSei09igSIFT.pdf)
      • Alpha channel, alternatives to contours RLE
      • New alpha transparency (per pixel) added
      • Introduce functions with advance ways of blending
      • Mask channels
  • 3D features
    • Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
    • PFH, FPFH from Radu,
    • 3D Delaunay,
    • 3D convex hull,
    • mesh generation,
    • mesh simplification/subsampling
    • Extend to using 2D XYZA grid as well as oct/kd trees
  • 3D capture
    • Probably by using silhouettes and then texture for cavities (?)
      • Possibly with textures or phased light
    • VSLAM
    • Image stitching
  • Optical flow
    • Variational optical flow
  • MLL
    • Tiechman’s
      • Kernel based gentle boost
      • Feature selection engine
    • Automatic parameter tuning
      • Cross-validation, ROC curve generator
  • High GUI
    • move to using QT as the HighGUI interface look and feel
  • Misc:
    • Focus detector,
  • Lighting/Specularity
    • Look into:
      • Using Specularities for Recognition, Margarita Osadchy, David Jacobs, Ravi Ramamoorthi (stored on my disk as specularities.pdf)
      • Retrieving Multiple Light Sources in the Presence of Specular Reflections and Texture_, Pascal Lagger and Pascal Fua (Specular_multiple_sourcesFua08a-1.pdf)
      • 3D Pose Refinement from Reflections_, Pascal Lagger, Mathieu Salzmann, Vincent Lepetit, Pascal Fua (Specular_laggercvpr08.pdf)
      • Detecting Specular Surfaces on Natural Images, Andrey DelPozo, Silvio Savarese
  • Histograms
    • Insert value function with interpolation
    • Automatic bin size adjustment for recognition performance using a test set
  • OpenCV Ideas 2009

Minutes

Release Schedule

  • Vadim going over bug tracker
    • Some are very old
    • Some new, more critical ones.
      • HighGui can’t be built on snow leopard
        • Quick solution is to disable those features on snow leopard
        • 3 parts: {X} interface, {OK} image i/o libjpeg, {X} video i/o
          • Possibly use GTK and ffmpeg for work around
          • Longer term: Use

IPP, MKL support

  • MKL was used only for matrix multiplication
    • For this release, we won’t restore
  • IPP is used different, before we detected IPP automatically. But now we use IPP like any other library.
    • Detect libraries at compile time using cmake scripts
    • Code is platform independent, but the cmake scripts support IPP only on windows
      • Need to extend to Mac and Linux
    • Will be done for 2.0
    • A lot of new code uses SSE in 2.0, so IPP is less important
      • FFT, polar, haar can still benefit cocoa

Matrix:

  • We aren’t going to use MKL, instead we’ll use the Free library called
  • Atlas
    • Why not Eigen? Atlas is BLAS interface, Eigen is a new API with templates etc. Atlas is lower level, so its faster

Bundle adjustment

  • Sergey said that he ran the tests, but it didn’t pass :-(
  • Sergey is in Moscow and will be back on Thursday.

User contributors

  • There’s a thanks page

64 bit support

  • OpenCV runs on 64 bit machines
    • It might not be stable over huge arrays since there have not been tests with this
    • Need bug submits from people who do this

Vadim:

The following has been done last week:

  1. The new C++ OpenCV documentation has been scanned another time, many formula and a few typos have been fixed. Bibliography has been fixed too.
  2. The HTML documentation can now be generated again, added a workaround for the problem with plastex.
  3. Checked the python bindings on Windows, they built well. Thanks to James for the quick fix.
  4. IPP support has been partially restored (now it’s detected on Windows only, but it’s easy to add Linux & Mac support too). The functions area covered are:

DFT (IPP’s DFT is still significantly faster than OpenCV’s), color conversion and Haar object detection.

  1. About 15 records in the bug tracker have been closed. Most of them are just obsolete, but some were real bug reports.
  2. Maria wrote a test for about half of the classifiers in MLL, all the tree based test
    • The test reads a database, trains, tests, saves the model, loads the model and retests. Scores should be the same. Rainer suggested this test as a sanity check.
  3. Anatoly has made good progress in the build system. We can build in different configurations, with different compilers gcc 4.1 through 4.3
    • Can enable or disable openmp and SSE
    • Will extend to Windows and Mac 0S 10
    • About 1/3 done
    • Test with ROS

The current post-beta activities:

  1. Continue to clean the bug tracker. Actually, out of ~140 bug tracker records approximately 50 or less are likely serious bugs that may affect many users. Most other bugs are obsolete, refer to the old-style Python bindings reports, or various feature requests, or very special issues with video capturing etc. So, it’s a good chance that the tracker will be cleaned by 2.0 release.
  2. Documenting the new cvaux stuff.

Plugs

Victor

  • Outlet detection_: the one way descriptor is used in the ROS outlet node now. A lot of time has been spent on setting up the online testing system. Finally I stopped on using prosilica_camera/fake_poll_node that simulates a stream of images from a camera with a set of images from a given folder. Another package, image_view, is used to generate images from bag files. One way descriptor takes unexpectedly long time (1-4s per frame) to classify keypoints in test images from new bag files collected by Jack. This is due to a large number of false alarms from the hole detector. ldetector works much faster (0.01-0.18s per frame) but gives more errors on white outlets (74 correct detections out of 100 as opposed to 90 with a GHT detector). The next step is to speed up 1NN in one way descriptor by using search structures. Also the outlet node crashes when processing 3D data from the outlet detector. Clearly it has to be generalized for arbitrary outlets (right now it assumes 2×2).
  • Plug detection_: a new method for small plug detection has been implemented. The method uses hole detector to detect black squares and then groups them assuming an affine transformation from the chessboard model to a scene. It detects all chessboard in data collected by Gary, all chessboards on white and black outlet data collected by Jack and about 50% of chessboard from orange outlet data collected by Jack. The latter is because of false alarms from the hole detector. The algorithm can be made more robust to such cases. The method is relatively fast — ~0.1s per frame. cvFindChessboardCorners as well as cvFindChessboardCornersex do not detect chessboards on these images at all.

Action Items

Gary

  • set up Thanks wiki after the meeting
  • Send Victor Jack’s code
  • Some test images to Victor on outlets

James

Vadim

  • Move THANKS to wiki
  • Get SVN commit statistics

Victor

  • Implement detection and tracking
  • Testing plug detection with new chessboard detector
  • Make sure outlet node doesn’t crash
  • Generalize for arbitrary outlets
  • Test for robustness
  • Talk to Brian about where to put data for cmake to pull in for the outlet detection node.

From Last time

Gary

  • Ask Marius about Chamfer
  • Ask about FLANN docs
    • There is sample code
  • Connect Victor and Jack

James

  • Work through some documentation examples for C++
  • Talk with Vadim about docs on line
  • Go through bugs, assign
  • Talk with Vadim about how to make Python and C++ interface more the same

Vadim * Talk to James about docs on line

  • Talk with James about how to make Python and C++ interface more the same

2009-09-28

Agenda

  • Release
    • Announcable schedule for 2.0?
      • Documentation?
        • Especially the C++ and FLANN
      • Bugs
  • Test system
  • IPP link issues?
  • Calondar Features
  • Jitendra’s N-Cut code
  • Bundle Adjustment status
  • Code change statistics
  • Some chat about post-release plans
    • Segmentation
    • 3D model capture
    • 3D support — working with Radu
  • NVidia, LRB optimizations
  • CVPR

Outlets and Doorhandles

  • Results on data that I sent right before ICCV

Schedule:

  • Beta 1.2.0 is released! Sept 14th
  • Gold release on September 29st, announce at ICCV 2009

For Gold 2.0

  1. Fix installation bugs
  2. Documentation
    • Fix formulas
    • Fix links
    • Improving references
  3. Anatoly and Maria are creating and running tests of OpenCV in various condigurations, OS et
  4. Bug tracker clean out

OpenCV Summer Release:

  • Dates:
    • {OK} Feature freeze July 31
    • {OK} Beta Released: Sept 14th.
    • Gold Release Sept 29, 2009
  • Keys:
    • {OK} Done; (./) In progress; :\ Um, maybe; {X} Left as an exercise for the user to implement

PLANS:

  • General Structure:
    • :\ (Probably past Beta) Provide way to build OpenCV statically, then the footprint will be variable and usually much smaller than now
  • New Features
    • {OK} C++ interface
      • {OK} cxcore
      • {OK} cv
      • {OK} HighGUI
      • {OK} I/O for sructures, histograms
    • Better python interface
      • {OK} In progress thanks to (James Bowman)
      • {OK} Move into OpenCV and deprecate SWIG only version → selected in CMake either or both
      • {OK} Use this to expand unit test functions for OpenCV
    • 2D features
      • {OK} HOG
        • {X} Chamfer matching with orientation In progress thanks to Marius Muja
      • (./) Keypoints (mostly thanks to LePetit group’s great work in this area)
        • {OK} LePetit keypoint detector YAPE * (Yet Another Point Extractor). We have found this to be the most stable detector.
        • {X} C-descriptor … (Vadim Michael — on site. Difficulties reproducing results)
        • {OK} Fern based planar patch detector from LePetit’s group D
        • {OK} LePetit one way descriptor (Victor) D
      • {OK} MSER — list of contours … how to extract keypoint from this
      • {OK} Fast detector *
      • {OK} SURF * D
      • {OK} Star *
      • {OK} Self-similar detector
    • 3D features
      • {OK} Bundle Adjustment (by Phase Space / Sergey)
      • {X} normals in grid (Gary)
      • {X} normal based 3D patch (Gary)
      • {OK} SPIN
    • Optical flow
      • {OK} Dense (Farnerback)
    • MLL
      • {OK} Extreme trees
      • {OK} Reduce memory footprint
      • {OK} More convenient training via a data class
      • {X} Nister trees (Patrick) Post Beta
      • {OK} FLANN or appropriate parts (Marius)
      • {OK} Generalized (boosted) cascade (generalizing Haar cascade — not Haar or LBP features)
  • Misc:
    • {OK} adaptive skin tracking
  • Documentation
    • {OK} Move to one Latex source
      • {OK} Use to generate HTML, PDF
    • {OK} Add search function
    • {OK} Add user feedback/documentation bug system
    • (./) Proof read — Needs complet re-do
    • {OK} Put up
      • {OK} cxcore
      • {OK} cv
      • {OK} MLL
      • {OK} Aux
      • {OK} HighGUI
    • (./) ’C++ Interface
      • {OK} C++ cxcore
      • {OK} cv
      • (./) Aux nothing for new functions
      • {OK} HighGUI
      • (./) Needs improvement on class descriptions
    • {OK} Create Python documenation
  • High GUI
    • {OK} In memory .jpg, .png
    • {OK} move to C++ interface
    • {X} visualizations, statistical data extensions
  • Testing functions
    • (./) Regression tests for all functions at least in python→OpenCv to make these easier to write
    • {OK} Bought new server for automated tests and builds
      • (./) creating auto test system → Usable results 3 weeks
  • (./) Define user contribution process:

Put off to Future

  • 2D features
    • Scale Invariant Feature Transform with Irregular Orientation Histogram Binning_, Yan Cui, Nils Hasler, Thorsten Thorm ?hlen, Hans-Peter Seidel (Feature_Point_Descriptors_SIFT_ImprovementCuiHasThoSei09igSIFT.pdf)
      • Alpha channel, alternatives to contours RLE
      • New alpha transparency (per pixel) added
      • Introduce functions with advance ways of blending
      • Mask channels
  • 3D features
    • Plane fit (delayed for integration with Radu’s point cloud library — more comprehensive approach)
    • PFH, FPFH from Radu,
    • 3D Delaunay,
    • 3D convex hull,
    • mesh generation,
    • mesh simplification/subsampling
    • Extend to using 2D XYZA grid as well as oct/kd trees
  • 3D capture
    • Probably by using silhouettes and then texture for cavities (?)
      • Possibly with textures or phased light
    • VSLAM
    • Image stitching
  • Optical flow
    • Variational optical flow
  • MLL
    • Tiechman’s
      • Kernel based gentle boost
      • Feature selection engine
    • Automatic parameter tuning
      • Cross-validation, ROC curve generator
  • High GUI
    • move to using QT as the HighGUI interface look and feel
  • Misc:
    • Focus detector,
  • Lighting/Specularity
    • Look into:
      • Using Specularities for Recognition, Margarita Osadchy, David Jacobs, Ravi Ramamoorthi (stored on my disk as specularities.pdf)
      • Retrieving Multiple Light Sources in the Presence of Specular Reflections and Texture_, Pascal Lagger and Pascal Fua (Specular_multiple_sourcesFua08a-1.pdf)
      • 3D Pose Refinement from Reflections_, Pascal Lagger, Mathieu Salzmann, Vincent Lepetit, Pascal Fua (Specular_laggercvpr08.pdf)
      • Detecting Specular Surfaces on Natural Images, Andrey DelPozo, Silvio Savarese
  • Histograms
    • Insert value function with interpolation
    • Automatic bin size adjustment for recognition performance using a test set
  • OpenCV Ideas 2009

Minutes

  1. release:
  • Windows binary doesn’t run on windows machine unless Visual studio is installed.
    • Since they used manifest
    • Mark might know how to fix
  • Windows should be out today
  • Linux out today or tomorrow.

Test build

  • Test build system is now working for Linux
  • Anatoly is working on the windows version running in Virtual box
  • Test data: Machine learning has more test code.
    • Will probably use python

IPP link was broken in Beta, but now its fixed in Gold

CUDA or Larrabee. Instead of accelerating already fast functions. Would be better to do

  • Graph cut stereo
  • Dense optical flow
  • Presence of GPU or LRB can be detected at run time and then called

Segmentation algorithms

  • Watershed segmentation (in, and fast)
  • Mean Shift filter — produces posterization
  • N-Cut Malik said he would donate this code.
  • Felzenschwalb fast algorithm

Calondar’s algorithm

  • Currently, outputs non-compressed signatures
  • Michael said he’s willing to help

Bundle adjustment

  • Still the way it was. Have tests, have code, but the test do not pass
    • Fix this after the release
  • Based on sparse Levenberg–Marquardt solver
    • Need to be able to compute the reprojection errors
    • and form the Jacobian

Code change statistics

  • From SVN

Documentation:

  • James rerun script for FLANN

CVPR tutorial?

  • OpenCV on Robots and/or robot simulators?

Vadim on vacation until Oct 1st – October 18

Future

  • Support for numpy
  • Support for the C++ interface
  • MLL

Outlet detection:

  • Trying to implement a system in ROS, bag file is played where a plug is inserted into an outlet. Calibrate so that distance is minimum.
  • Learning ROS technical details
    • Plug detector uses the new chessboard detection algorithm
    • Pose of plug looks reasonable
  • Outlet detector has a bunch of false detection on hard to observe images
    • Right now, filter out cases by scale selection
    • Orange outlets, detection is quite stable
    • White outlets are hard due to low contrast
      • Classify features using 1-way descriptor on several scales
      • Then run generalized Hough transform over multiple scales against the template
        • When contrast is low, power hole can be classified as small ground hole due to bad scale
      • Solution is to search on scale first.
    • Plug detector is working with the new algorithm

Vadim

  • OpenCV 2.0 for Windows and Linux/Mac has been released, announcement was posted to yahoogroups.
  • 60 bug reports have been moved to pending state, mostly with the resolution “fixed”. Currently there are about 80 opened bugs.
  • vs.net2003 support added, fixed build on MacOSX 10.6 with default compiler (thanks to Mark Asbach for helping me)
  • good progress on the build system has been made. It builds opencv on 64-bit linux with 3 compilers, debug/release, sse on/off, openmp on/off. In each configuration the tests are executed. All the results can be viewed online, the link will be sent separately. Now Anatoly is working on adding in the windows part

Victor

  • Outlet detection: A filter for partially observed outlets has been implemented. With a forearm camera a partially observed outlet is a common situation so we have implemented a filter that reduced the amount of false detections. The outlet detection algorithm assumes that a part of an outlet tuple can be occluded and deals with these situations correctly the most of the time (e.g. results on specular dataset). However, if only 2 out of 4 outlets are visible, detection is imprecise, so we are filtering these cases out.
  • Outlet templates downloading has been automated. A tar.gz archive is downloaded and unpacked when outlet_detection package is built.
  • Plug detection: a new plug detection function has been plugged into the plug_node. Plug detection has been tested online using fake_poll_node, the plug is detected correctly the most of the time and its pose is reasonable. I am working now on a final test that will calculate the difference between outlet hole pose and plug tip pose. The success criterion is a small distance between the two when the plug is inserted into the outlet.
  • Gary, I’ve implemented automated download of outlet templates on the built stage of the outlet detection package. Right now the archive can be downloaded from http://itseez.com/data/outlet_templates_20090928.tar.gz. Could you please place it somewhere in ipr:/var/www/pr.willowgarage.com/data/outlet_detection?

Action Items

Gary

James

  • Rerun script so that FLANN documentation shows up

Vadim

Victor

From Last time

Gary

  • set up Thanks wiki after the meeting
  • Send Victor Jack’s code
  • Some test images to Victor on outlets

James

Vadim * Move THANKS to wiki

  • Get SVN commit statistics

Victor

  • Implement detection and tracking
  • Testing plug detection with new chessboard detector
  • Make sure outlet node doesn’t crash
  • Generalize for arbitrary outlets
  • Test for robustness
  • Talk to Brian about where to put data for cmake to pull in for the outlet detection node.

2009-10-06

Agenda

Vadim is out on post release vacation

Gary at ICCV and TUM on thesis committee

No meeting except for *Victor*’s progress update on outlet detection

Minutes

Victor

Outlet detection: a new algorithm for scale-space search has been implemented. Now all keypoints are classified (into ground holes/power holes/background) in the same scale, the same scale is applied to a template during GHT search. Iteration through scales is an outer cycle of the algorithm. As a result accuracy on orange2x2_gain64_exp066_30hz test dataset has grown from 15 correctly detected images to 32. Also, the algorithm is more stable to occlusions, allowing correct detections even with a plug in front of the outlet (see attached image).

Filtering of inconsistent outlet detection results is implemented. GHT returns very rough positions of the holes and we choose the closest detected hole for each of the returned ones. However the closest hole can be a misdetection (for instance, a black circle near the ground hole on 2×2 fake wall orange outlet). The current algorithm calculates an error between the detected holes and the corresponding holes of a template projected with an affine transform. If error is large enough, positions returned by hough are used.

Outlet node is tested together with plug node for outlet/plug pose consistency. After minor modification, rviz shows consistent behavior of 3D positions of the plug and the hole that the plug is inserted to. Right now this was tested on orange2x2_gain64_exp066_30hz only, plug node still has to be generalized for 2×1 outlets.

Plug detection: the new chessboard detection algorithm has been tested on outlet_fuzzy_camera dataset. 850 images have been sampled from the orange outlet bag file. A plug is detected on 560 of them.

calonder_descriptor ROS package has been ported to mac.

Action Items

Gary

James

Vadim

Victor

From Last time

Gary

James

  • Rerun script so that FLANN and C++ documentation shows up

Vadim

Victor

2009-10-13

Agenda

Minutes

_’Outlets_’

  • 0.5 – 1 second for outlet detection
  • Plug pose is real time
    • “out of the box” it worked 25% of the time on new plug (new forearm camera has a lot of noise).
  • Right now the outlet and plug nodes work with prosilica camera as a service — send a request and get a response back
    • Need same mechanism with forearm camera

_’CVPR_’

  • Could the outlet detector be a paper?
  • Fusing edges and local descriptors features

Vadim

  • Out of town

Gary

  • I moved the outlet templates

Victor

  • Outlet detection: a node that allows testing outlet and plug detection has been implemented. It simulates prosilica_poll service from a set of jpg files, just like fake_cam_node in prosilica_camera package, but it issues each image exacly two times, making sure it received outlet and plug poses for each image. As a result we are able to calculate distance between outlet and plug images synchronously. The new node, fake_poll_node_sync, is placed into outlet_detection package.
  • Plug detection has been improved by recalibrating the plug on the new data (calibration finds the relative position between the chessboard and the tip of the plug).
  • Outlet detection has been speeded up from 3s per frame down to <1s per frame by selecting ROI based on the previous frame. Features outside of ROI are filtered out and they do not participate in one way descriptor classification. The latter is still the bottleneck of the method. We are experimenting with ferns (ldescriptor) and calonder descriptors. Ferns shows worse results compared to one way even after parameter tuning, results on calonder descriptor are on the way.

Anatoly

  • parallelized the buildbot for linux, now different configurations can be built in parallel. A lot of time was spent on Windows OS: cmake produces various errors when started from python (no problems in cmd shell). This is still in progress.

Maria

  • studied graph cuts and segmentation based on graph cuts as well as ways of reusing graph cuts implemented in opencv.

Action Items

Victor

  • Send examples of forearm camera before and after pictures showing noise.
  • Work on stability of plugs and outlets with new data
  • How to work with forearm camera nodes

Gary

  • More data from point of contact of plug with outlet
  • Arrange working visit from Victor
  • Renewal decision

James

Vadim

2009-10-21

Agenda

  • C++ Documentation
  • OpenCV Code Reivew
  • Outlets
  • Future direction brainstorm

Minutes

  • Documentation changes
    • Make C++ primary
      • Keep C but reference the C++ for big descriptions
    • Vadim will start with CV module and see how it goes. Probably 2 weeks
      • Most CV is done 90%
      • Whole cxcore
      • None of cvaux
      • All of HighGUI
  • Information on relative speed-ups for function optimisations

FUTURE DIRECTIONS BRAINSTORM

  • General Idea:
    • Build a tool chain
      • Implement higher level functionalities
      • Find out why it “took so long”
      • Fix the gaps
      • Iterate to next higher level functionality
  • Python wrapper covering all of OpenCV
  • Get Calander descriptor in! Almost there.
  • Getting the C++ docs out
  • Higher level algorithms for complete solutions
  • Collage/stitching
  • Image stitching
  • Working with humans — face, hands, skins, finding in 3D (do with single camera)
    • Face recognition
    • Indoor people recognition
  • Time for C++ interface to be more consistent
    • Point classes — some operators that are missing (can multiply but not divide)
  • OpenCV 2.0 “new features detectors and descriptors”
    • OpenCV 3.0 “new object detectors and 3D functionality”
      • Face rec, people
      • Project with outlet detection
  • 3D descriptors
  • Open alternative to SIFT
    • Definitively have an alternative to SIFT/SURF for keypoints on tracking and on recognition scores
  • Completing Segmentation toolbox
  • Visual odemetry, visual slam (from Kurt and James?)
  • Performance — acceleration opportunities — GPUs via OpenCL
    • Graph cut, fast stereo, fast HOG
  • Object recognition with sparse stereo data (Paper Silvio forwarded)
  • Collaboration with Bernt Scheile ?
    • Model based vision
  • 3D rendering for 2D and 3D visual training
  • 3D model fits (Move 2D and 3D chamfer matching into OpenCV from Marius)
  • 3D model capture
  • Lighting perception — specularities, direction of lighting, shape from shading and shape from texture
  • flexible objects
  • transparent objects
  • surface recognition (wet, wood, cement …)
  • Collaborate with Sid’s toolkit?

Papers to Consider

  • Object Recognition and full Pose Registration from a Single Image for Robotic Manipulation
    • Collet, Berenson, Srinivasa, Ferguson
  • A Probabilistic Framework for 3D visual Object Representation
    • Detry, Pugeault, Piater
  • Latent SVM
    • Object Detection with Discriminatively Trained Prt Based Models
      • Felzenszwalb, Girshick, McAllester, Ramanan
        • Code: http://www.cs.uchicago.edu/~pff/latent
  • Appearence-based Keypoint Clustering
    • Estrada, Fua, Lepetit, Susstrunk

Outlet Progress

  • Images are low contrast
  • Low exposure
  • Working to improve the results across conditions
  • Working on using lines instead of keypoints
    • Show working on European outlets

Weekly reports

Victor

  • Outlet detection: images extracted from the latest dataset plug_touch_2009-10-17-21-21-33-topic (using image_proc) are considerably lower quality than the images from previous forearm datasets. Outlets are not detected robustly because of low contrast holes. Accuracy is about 50%, however many false alarms can be filtered out based on aspect ratio and scale. We did not do this filtering before because we meet such issues for the first time.
  • A new version of outlet detector where keypoint classification is done with calonder descriptor instead of one way descriptor is implemented. Classification accuracy depends on parameters of calonder descriptor, outlet detection results are comparable to one way descriptor on the older datasets (orange2x2_gain64_*). We expect calonder_descriptor to work much faster than one way descriptor.
  • Outlet node has been updated to be compatible with the latest ROS rc.
  • Plug detection: the new chessboard detection algorithm has been improved by a better corner ordering. Plug is robustly detected on the plug_touch_2009-10-17-21-21-33-topic dataset.

Vadim

  1. 5 bugs in OpenCV 2.0 have been fixed.
  2. started collecting statistics and sample conversion to the new interface:
    • Converted samples from the OpenCV book ch2 (100%) and ch3 (50%). The archive is attached.
    • Here is the statistics collected:
      • the number of functions converted from C to C++: 401 (note that some functions, like cv*PCA*, or cv*SVD*, or cv*SURF* etc. have been converted to C++ classes, so the real number of the functions is slightly bigger, somewhere in the range 400-420)
    • the number of the optimized functions:
      • ~25 (cxcore) + 33 (cv) + 2 big classes in cvaux (spin images & HOG).
      • Given that some of the functions (like cvFilter2D etc.) include many flavors, the real number of optimized internal functions is likely over 100.
      • Statistics for the year 2009:
        • SVN: read transactions >130K, write transactions ~760K, files modified ~3400.
        • bug tracker activity (number of records opened/closed): 359/318
        • number of downloads: 389000.
      • Statistics for the year 2008:
        • SVN: 6/50/450 (since the migration only happened in December); the statistics on CVS has been lost.
        • bug tracker: 87 opened/27 closed
        • number of downloads: 476000
  3. Anatoly continues working on the build system. Last week he added
    • full Windows and 32-bit Linux support (where the virtual Windows machine is loaded, the buildbot service is automatically executed, connected to the master and starts building and testing of OpenCV on Windows in different configurations).
    • Because of various problems with Virtualbox and VMWare (low performance) it was decided to migrate to MacOSX as the host platform and use VMWare Fusion. The migration was successfully completed.
  4. Maria is implementing GraphCut, >50% of the code is written.

Action Items

Gary

  • Prepare review foils
  • Set up a review date
  • Send a categorised brainstorm list

James

Vadim

  • Make standalone version of C++ docs
    • Make depecation statement for the C docs and list of C++ only functions
  • Send foils for OpenCV review
  • Send before/after optimization information

Victor

  • Send additional foils for outlet if you think they will help

From Last time

Gary

James

Victor

2009-10-27

Agenda

  • Review
  • Brainstorm
  • Outlets

Minutes

  • Previous Statistics
    
    
  • Here is the statistics collected:
  • the number of functions converted from C to C++: 401 (note that some functions, like cv*PCA*, or cv*SVD*, or cv*SURF* etc. have been converted to C++ classes, so the real number of the functions is slightly bigger, somewhere in the range 400-420)
  1. the number of the optimized functions:
  • ~25 (cxcore) + 33 (cv) + 2 big classes in cvaux (spin images & HOG).
  • Given that some of the functions (like cvFilter2D etc.) include many flavors, the real number of optimized internal functions is likely over 100.
  • Statistics for the year 2009:
    o SVN: read transactions >130K, write transactions ~760K, files modified ~3400.
    o bug tracker activity (number of records opened/closed): 359/318
    o number of downloads: 389000.
  • Statistics for the year 2008:
    o SVN: 6/50/450 (since the migration only happened in December); the statistics on CVS has been lost.
    o bug tracker: 87 opened/27 closed
    o number of downloads: 476000
  • _’Review_’
    • Start times
      • 2008, May Vadim
      • 2008, Dec Victor
        • 2009 later for rest of team
    • Statistics
      • Increase in code/functions
        • MLL (Cascade, FLANN)
        • Specific functions
          • stereo, HOG, Self similar,
          • ’OpenCV 2.0 Change log
      • Popularity
        • SVM (source forge.net opencv library (helpful reviews)
        • User group
        • Book
    • Code examples
      • Internal OpenCV: code (how the C++ interface is done)
      • Use of OpenCV: Write same example use of OpenCV in
        • C and
        • C++
        • Python
    • Demos
      • Plug and outlet
        • ’Plan A‘: Use Jack’s data
        • ’Plan B’: Set of files captured from BAG — stream from prosilica pose and outlets
        • ’Plan C‘: Make outlet with imagery that looks like Jack’s data
      • Examples of C vs C++
        • Examples of first 2 chapters
      • Show python
        • New wrappers are from James
        • Move to C++ — Vadim
        • Old style swig wrappers
        • Face detection (python)
      • C++
        • People detection for HOG
    • Future
      • Make ~mobil eye/Felzenschwalb Machine set up — parts-whole meta-functionality
      • 3d object recognition package
      • NVidia
        • CUDA
        • OpenCL
      • Version 2 of book
  • Previous Brainstorm
    			
    • General Idea:
      o Build a tool chain
      + Implement higher level functionalities
      + Find out why it “took so long”
      + Fix the gaps
      + Iterate to next higher level functionality
    • Python wrapper covering all of OpenCV
    • Get Calander descriptor in! Almost there.
    • Getting the C++ docs out
    • Higher level algorithms for complete solutions
    • Collage/stitching
    • Image stitching
    • Working with humans — face, hands, skins, finding in 3D (do with single camera)
      o Face recognition
      o Indoor people recognition
    • Time for C++ interface to be more consistent
      o Point classes — some operators that are missing (can multiply but not divide)
    • OpenCV 2.0 “new features detectors and descriptors”
      o OpenCV 3.0 “new object detectors and 3D functionality”
      + Face rec, people
      + Project with outlet detection
    • 3D descriptors
    • Open alternative to SIFT
      o Definitively have an alternative to SIFT/SURF for keypoints on tracking and on recognition scores
    • Completing Segmentation toolbox
    • Visual odemetry, visual slam (from Kurt and James?)
    • Performance — acceleration opportunities — GPUs via OpenCL
      o Graph cut, fast stereo, fast HOG
    • Object recognition with sparse stereo data (Paper Silvio forwarded)
    • Collaboration with Bernt Scheile ?
      o Model based vision
    • 3D rendering for 2D and 3D visual training
    • 3D model fits (Move 2D and 3D chamfer matching into OpenCV from Marius)
    • 3D model capture
    • Lighting perception — specularities, direction of lighting, shape from shading and shape from texture
    • flexible objects
    • transparent objects
    • surface recognition (wet, wood, cement …)
    • Collaborate with Sid’s toolkit?

Papers to Consider

  • Object Recognition and full Pose Registration from a Single Image for Robotic Manipulation
    o Collet, Berenson, Srinivasa, Ferguson
  • A Probabilistic Framework for 3D visual Object Representation
    o Detry, Pugeault, Piater
  • Latent SVM
    o Object Detection with Discriminatively Trained Prt Based Models
    + Felzenszwalb, Girshick, McAllester, Ramanan
    Code: http://www.cs.uchicago.edu/~pff/latent
  • Appearence-based Keypoint Clustering
    o Estrada, Fua, Lepetit, Susstrunk

Vadim

  • Over 10 bugs in OpenCV 2.0 have been fixed (http://sourceforge.net/tracker/?limit=75&func=&group_id=22870&atid=376677&assignee=&status=4&category=&artgroup=&keyword=&submitter=&artifact_id=&assignee=&status=4&category=&artgroup=&submitter=&keyword=&artifact_id=&submit=Filter + a few bugs not in the list). these are various fixes for build problems, test crashes, incorrect functions behaviour in some cases etc.
  • C doc has conditionals for Python
    • One set of docs that is all conditionals
      • C++ is separate since Vadim didn’t want to break
        • Conditionals should go into the C++ docs
  • Graphcut draft version is ready, not working well yet (as usual), Maria is debugging it now
  • Build system on Windows has been debugged, MacOSX added.

Outlets

Victor

  • Outlet detection: GHT algorithm has been improved. The old algorithm allowed voting of several keypoints for the same position and the same pose. As a result we could get 12 votes for an outlet pose from 12 keypoints with the same coordinates. In practice we were getting lots of ambiguous poses. This is prohibited now, and the number of votes for a specific pose can’t be larger than the number of holes (12 for 2×2 outlet).
    • Calonder descriptor has been integrated into outlet detection as an alternative for one way descriptor for faster classification.
      • The calonder results are worse but we didn’t finish tuning the parameters. The final test results are on the way.
  • The literature research on possible ways of simultaneous tracking of points and lines is done. There are two possible ways:
  • use edges for voting within generalized hough transform;
  • use geometric hashing for pairs of keypoints and edgels.
  • Current results
    • cracked (correct/total): 100/100 | Average time: 0.622961
    • fake_wall_2x1white (correct/total): 109/111 | Average time: 2.145840
    • fake_wall_2x2orange (correct/total): 34/34 | Average time: 2.277736
    • orange2x1 (correct/total): 99/103 | Average time: 0.535686
    • specular (correct/total): 20/56 | Average time: 0.949764
    • white2x1hall (correct/total): 103/103 | Average time: 0.489211
    • white2x2 (correct/total): 102/104 | Average time: 1.026165

Action Items

Gary

James

Vadim

  • Send speedups

Victor

From Last time

Gary

  • Prepare review foils
  • (./) Set up a review date
  • Send a categorised brainstorm list

James

Vadim

  • Make standalone version of C++ docs
    • Make depecation statement for the C docs and list of C++ only functions
  • Send foils for OpenCV review
  • Send before/after optimization information

Victor

  • Send additional foils for outlet if you think they will help

2009-10-29

Agenda

Documentation

Minutes

  • Large amount of documentation
  • Probably 2 weeks just to do anything to the whole corpus
  • Need to merge a single document with the APIs in
    • Vadim: C document will not change much
      • Won’t change much
      • Main reference manual would be shorter with less conditionals
    • James:
      • Uses C document a lot and there’s lots of document bugs there (bugs, broken links, inaccurate text, missing info)
      • Don’t want to freeze this as is
  • V: Put most of things to new document, old API could mostly link to the new document
  • J: That’s one way of having “one” document, but bad for users
  • V: Old style interface is getting obsolete
  • J: 3.0, C is deprecated
  • V: Old API will stay past 3
  • J: Was using cvCalibrateStereo
    • A lot of this document is common to C, C++, Python
    • meaning of enums etc. Everyone will want to know common
    • value of the function description is >> call signature
  • V: function descriptions can stay, but formulas, descritions can
  • J: This will be a pain for users to keep multiple links
  • V: Documentation can get quite big
    • Argus is switched to new API
    • Would just do good documentation for new API
  • J: Then what’s the plan for C documentation?
    • V: Function and parameter descriptions as is, rest point to new API in C++ reference
  • J: Most of the complexity is in the parameter descriptions. Where would these description reside
    • V: feature not bug, because if C++ extends API, then it can change and C doesn’t change
    • J: Bugs in documents as well as C
  • V: Most users will go to new API in 1-2 years
  • ’Issue:
    • 2 docs (C++ and C) vs a merged one with conditionals
      • Advantage is simpler main doc
      • Disadvantage is maintain 2 version
  • Maintaining semantics
    • New API may get extra parameters
  • But, lots of work needs to be done to improve the documentation
    • So its not the case that the C document can be locked down
    • Might be many 1000’s of changes to the documents
      • That gets hard with 2 documents
  • Main issue is duplication: something in 2 places is harder to maintain
    • Wiki and HTML slowly drifted, want to avoid this drift again
    • CxCore, CvRef — all in big files, maybe if these are split into smaller blocks
      • Camera calibration for example, the API is roughly the same between C and C++
      • Feature descriptors are only C++
    • Maybe splitting the documents that are mostly different, but if only parameter differences, then makes sense for 1 function
    • For example, HOG is just in C++
    • Pulling new functions into C++/Python only files
  • V: CvAdd has large differences, allows arbitrary. C is overloaded. Default parameters
    • In C++ can’t default references This causes differences. Python uses output tuples

Platex is quite fragile and so we need to make sure that documents

Decision

  • We should get someone
  • We should bring the Python and C++ together.
    • Numerical python … use their numerical types so that we can make use of python’s optimization etc functionality
    • Can replace a lot of cxcore
  • MLL (C and C++) can be done by Maria
  • Cluster docs into join and separate groups
  • Merge functions listed in *Vadim*’s email (functions with significant differences)
    • This will set the style
    • Someone else do this for the rest
  • Make sure the latex documents still builds

Action Items

Gary

  • Find someone who can do latex to merge and divide C and C++
    • Produce a file with conditionals for all 3 target languages

James

  • Take functions Vadim listed (that have significant API differences C vs C++) to set the style of how to document

Vadim

  • Cluster the doc files into “Conditionals” and “Separate” files
  • Document the undocumented new functions

2009-11-02

Agenda

OpenCV Review

  • Code examples C vs C++
    • Internal (before and after)
    • External (such as writing a focus detector old-style, new-style
  • Demos
    • Python person detector
    • C++ … ? (dense optical flow)
    • Outlet detector
  • Speed up statistics
  • Template compatible/STL/Boost
  • Testing

Minutes

Vadim

  • Completing work document on OpenCV members
  • Victor on himself and Alexi
  • Prepared spreadsheet on performance test results comparing 1.0 (IPP and without) to 2.0 (IPP and without)
    • A few foils of summary results
  • C++ Test coverage cxcore 80%, cv 55%, cvaux 0%, MLL 30%
  • Python Test coverage
  • Template
    • Internally OpenCV uses templates
    • Template over cv::Mat
    • Template is light
      • Can make a template cv::Mat with certain restrictions —
        • multi-channel OK
        • Every channel has the same type (8bit, integer, fp, 64) but no mix
        • OpenCV can take STL vectors of the above types

Demos

  • Person detector C++
  • Dense optical flow
  • Grabcut
    • Uses EM
    • Then graphcut
  • Outlet detection

2010

  • Improve documentation
    • Help to merge conditionals
    • Improve the content
  • Getting the test coverage up to 100% (probably mostly using python)
  • 3D capture
    • Robust segmentation
    • 3D for silhouette
      • Gives you a point cloud
    • Point cloud to mesh
    • Use textured stereo and do a global fit
      • With and without calibration pattern
  • 3D Model fitting
  • MLL
    • Latent SVM,
    • Better classification infrastructure
    • Something like IDEAL, or interface to R, Python
      • Object detection framework (visualization)
        • Use package for CRF
    • 2D and 3D features engines
  • Image collage for Google summer of code
  • Image stitching (subset of Visual Odemetry/VSLAM)
    • Alternatives for SIFT/SURF
  • Tracking toolkit

GPU Acceleration NVidia

  • Super resolution from video (interframe interpolation)
  • Latent-SVM + HOG
  • Stereo and energy min stereo

Outlet Detection

  • Outlet detection: outlet node has been generalized to handle 2×1 outlets.
    • Experiments with white2x1_gain64_exp033_30hz have shown that pose estimation is less stable compared to 2×2 outlets. Also,
    • when a plug is inserted into an outlet, detection disappears because 50% of the structure is occluded.
    • Outlet and plug nodes have been adapted to simplify their demonstration.
    • A new version of outlet templates is prepared with several bug fixes.
  • Outlet detection has been improved by adding chessboard filter:
    • features that belong to chessboard are not used for outlet detection and this increases detection rate for cases when a plug is close to an outlet.
  • The first version of edges tracking based on geometric hashing algorithm has been implemented and is being debugged.
    • Interpretable results are expected on Th.

Action Items

Gary

  • (./) Ask James about Python test coverage
  • (./) Finish the ppt presentation
  • (./) Finalize *Victor*’s Outlet detection trip

Vadim

  • (./) Send speedups
  • (./) Example of use of STL in C++
  • (./) Send some person detector images
  • (./) Account for Argus times

Victor

  • (./) Get working example of outlet detection to Gary
  • (./) Work on high angle/low contrast outlets
  • (./) Account for time spent

2009-11-09

Agenda

Discuss next year’s organization and priorities:

  1. The Willow OpenCV committee wants more visibility into who is doing what. More people should work in an SVN branch and then Vadim or other can merge to trunk.
    • Victor notes that many of his and Alexey’s committes were in ROS around outlet detections
    • I also think it would be good when people start working, they either log into our wiki (I could set up a monthly page) and just put a line about what they are doing: “optimizing GrabCut, fixing documentation for Calibrate, …”
      1. Maybe we should use Twitter for this? We could make a joint account and just twitter a line.
      2. James wants to know if someone is working on something he is using
    • We can actually use our Hudson computer to run test code whenever there is a checkin and send email to the offenders. I think we definitely should do this (even if you also do it locally).
  2. James and Brian want more effort on test, bug fix and documentation
    • James questions whether Vadim should be the main bug fixer or test cases for older functions, that someone more Junior should do this full time, continuously.
    • Steve: Once more visibility into who is doing what is established, It’s possible that maybe we should add low level people to code maintenance to complement the dual purpose of advanced code development.
    • Brian said that, like ROS, the core, highly used functions should have complete code coverage and good documentation, but other functions can be less strict.
  3. Brian and James are setting up Bullseye — a code coverage reporter. I think they will set it up to run once a day and you’ll be able to check code coverage statistics.
  4. Going forward, the development methodology should be that no new function should be added without
    1. documentation
    2. unit tests
    3. Unit tests can be in Python, it might better if they are in C++ though
    4. Unit tests (doesn’t matter, C++ or C) can/will be set up run on every new checkin.
    5. We’ll need in OpenCV’s case to set this for 32 and 64 bit systems as well as under different OS’s
  5. Steve wants to get back to a release every 6 months even if it means limiting what goes in. He understands that there was a lot of re-org going on this year, but he wants to get to that. If 6 months just doesn’t make sense, then we can propose something different.
  6. In light of the above, most current work (except for finishing off outlet) should go to test system and docs
    • we’ll want to show good improvement in coverage over the next month. I can help some in this once I see how a test should be written in C++ or Python.
    • We can consider hiring a person to get the docs merged into a single doc relatively quickly … I might do that on our side.
    • I wonder if we should put in speed tests into each test so that we can just read off the optimization improvements?
  7. Our areas for next year are still:
    • Object Recognition and Pose Toolkit
    • 3D Toolkit (Pretty much Radu will drive this … or not — off the list)
    • 3D Model Capture
    • Visual Odometry (James will drive this … or not — off the list)

Minutes

Moving SVN

  • How to do? Brian, Jeremeny, James or Scott?
    • Looks like Brian will do this, just got an email

Going forward, every new function must have

  • Documentation in Latex
  • Test code

Bugs

  • 130 open bugs, some duplicates
  • Before 2.0, tried to fix the most critical
    • About 80 of the bugs were about SWIG which will be obsolete
    • Some are old
  • Bugs are better fixed by the author of the function
    • Low level bugs can be hard in someone else’s code

Test

  • This is good for less senior people
  • Keep people assigned to areas that are important and that they become expert in
  • Code coverage tool needs to be run while working on to make sure you hit the branches
  • Prioritize by most used functions

Documentation

  • Everyone at Argus can write and read English
  • Less senior people can also do this
  • Vadim needs to pre-cluster the docs into language specific and all language
    • Then perhaps Anatoly can put in the latex conditionals
    • Vadim to do the initial clustering
  • Anytime a test is done, docuementation should be clarified
  • Every function has to have I/O type specified
    • In place operation or not
  • Prioritize by most used functions

Attaching Machine learning to an external toolkit (Sage, Orange, R?)

  • Fern as good as Calandor in accuracy
    **

Put Calander in as first as an example of going forward.

  • With good docuementation
  • With test code
    • Alexey has been working with this on the outlets

Vadim

  • 2 new opencv Python demos (people detection, dense optical flow), 3 C++ demos (dense optical flow, grabcut, convex hull) have been added, people detection demo in C++ has been extended to work with a list of images.
  • opencv C++ test system has been extended to display random seed values right in the console, which will help to reproduce test crashes.
    Accuracy of the functions’ speed estimations has been improved: on MacOSX mach_absolute_time is used; the new cv::getCPUTickCount() reads CPU timestamp on x86, x64 and ppc architectures. Median of the several measured function executions is selected.
  • GrabCut – Maria has rewrote OpenCV maxflow code in C++, where the edge weight can now have an arbitrary type. This new maxflow code is now used in GrabCut, so we have a complete implementation of the algorithm.
  • Started experiments with numpy. For now we do not subclass numpy’s nd-array (perhaps, we can do that later right in Python), but use it as-is.
    - Numpy does not support multi-channel arrays directly (one can make an array with elements of custom type, but this is very inefficient, and incompatible with OpenCV). Instead, it is possible to emulate multi-channel arrays using 3-dimensional arrays, e.g.: 320×240 3-channel image can be represented by 240×320×3 array. The layout of such arrays will be absolutely compatible with OpenCV, and the element access will also be very convenient.
    - Experimental implementation of cv.create (counterpart for cv::Mat::create, the key method in the C++ API) has been written.
    The C++ OpenCV API style is when you pass the destination image into the function: cv::erode(src, dst);
    The Python/numpy style is when you create the destination array and return it: dst = erode(src).
    The proposed OpenCV Python API style is to add “dst” keyword with default “None” value (def erode(src, dst=None)):

real_dst = erode(src[, dst=…]) # usage

- If dst is passed and it is a matrix of the proper size and type, it is reused (no reallocation is happened)
- If dst is a matrix of different size or type, or it’s not specified, when the new output matrix is created and returned.
This way we can decrease the footprint, do in-place processing when needed, and in general provide
more convenient Python-style API for quick experiments.
- cv.add with the proposed style has been implemented: def add(src1, src2, dst=None, mask=None).

Victor

  • Outlet detection: the first version of affine-invariant joint detection of edges with keypoints has been implemented. The algorithm constructs affine bases spanning all possible triplets of two adjacent edge points and one keypoint for each pair of edge and keypoint.
    • A 2D hash table (discretization of affine coordinates) contains links to corresponding model (pair of edge and keypoint) and basis if there is a point from the model that has coordinates in the corresponding region.
    • Recognition phase consists of building a voting histogram of bases (counting votes for specific bases from each of the test edge+keypoint pairs) and running a verification stage (match error calculation) for each basis that receives sufficiently high amount of votes.
    • Detection on orange2x2_gain64_exp066_30hz and white2x1_gain64_exp033_30hz sequences shows that oval edges around each outlet get consistently higher scores given a training edge from corresponding templates.
      • Keypoints in this algorithm are used only for constructing bases. We believe that after algorithmic optimization the algorithm will improve detection and pose estimation.
  • Outlet and plug detection demo preparation for OpenCV review.
  • Experiments with calonder descriptor instead of one way descriptor.
    • Calonder descriptor shows lower accuracy than one way descriptor even after parameter tuning.
      • It often classifies outlet holes as background causing outlet misdetections.
    • The next step in optimizing outlet detection is to implement kd-tree based NN search in one way descriptor.
  • Victor will be in the office on the 24th

Action Items

Gary

  • (./) Ask about how to upload SVN (Brian sent email, will do)
  • Ask about how Bullseye handles dynamically loaded code
  • Get a place for Victor in the Green room
  • Send Steve a note about contract renewal

Vadim

  • Pre-cluster the docs into language specific and all language
    • Then perhaps Anatoly can put in the latex conditionals
  • Make sure we migrate SVN and work with Brian to get test code coverage up and running

Victor

  • Send node about contract renewal
  • Have Alexey prepare Calandor feature for OpenCV inclusion together with
    • Documentation
    • Test code
    • Usage sample

From Last time

Gary

  • (./) Ask James about Python test coverage
  • (./) Finish the ppt presentation
  • (./) Finalize *Victor*’s Outlet detection trip

Vadim

  • (./) Send speedups
  • (./) Example of use of STL in C++
  • (./) Send some person detector images
  • (./) Account for Argus times

Victor

  • (./) Get working example of outlet detection to Gary
  • (./) Work on high angle/low contrast outlets
  • (./) Account for time spent

2009-11-17

Agenda

  • Code Coverage
  • Documentation advances
  • Code
    • Grab cut
    • Calandar descriptor
  • CVPR plans

Minutes

  • The way Brian set up code coverage mixes everything together, in particular cvaux (experimental code) so it doesn’t show progress
  • Get him to change the display

Code coverage

  • Probably 10% improvement to around 70% function coverage
  • Need to get Hudson to report by library categories

Documentation

  • Finished cxcore part of reference manual
    • It’s now clustered into universal and particular language differences
    • Conditional are in
    • It’s not up yet
  • Need to finish this with cv and with highgui
    • Can finish by next meeting so we get C++, then long merge to minimize duplication
  • mll — don’t know how to process it yet
    • 2.0, no C++ documentation for MLL
    • Also by next Tuesday

New MLL interface

  • For each train and predict, added new style matrices
  • Have to add these to the mll manual
  • C++ torch
  • Java Weka (is the most popular library right now)
  • R
  • Python?
  • Maybe the best way is to just write interface to spit out features on images setting locations/scales etc in different image locations
    • Output into their input framework
      • Feature production toolkit
      • OpenCV XML to HDF, Matlab, R
  • Might think of putting algorithms that are
    • Fast
    • Scalable
      • Extremely random trees and/or FLANN
    • Build features for this
      • Put together cluster here

GrabCut

  • In cv
  • No documentation
  • Doesn’t have test
  • Does has sample code

Focus detection

  • Function outputs Gaussian Pyramid
  • Turn into Laplacian
    • Upscale and subtract

Image pyramids

  • Have the original
  • Can do with smooth and resize
    • In IPP functions that combine both steps
      • smooth and decimate, only compute decimated pixels

Calandar Descriptor

  • Alexi could do it. Probably a week
    • Remove boost dependencies
    • Write test and sample (not in ROS)

CVPR

  • Tutorial deadline Dec 4th
    • Probably suggest with ROS/robot perception, OpenCV C++ overview
  • Workshop deadline Nov 22nd
  • Demos deadline April 13
    • We’ll do this for sure
    • Try to get NVidia involved or doing their own

Vadim

  • Migrated to the new SVN at code.ros.org
  • GrabCut API has been simplified (reduced to a single function). The code is now in SVN [Maria]
  • Wrote several new tests for OpenCV in order to improve the test coverage:
    - cxcore: 2 tests (RNG, XML&YAML I/O) [Vadim].
    - cv: 2 tests (inpainting, watershed) [Anatoly]
    - cvaux: 1 test (FAST corner detector) [Anatoly]
  • Fixed 4 bugs in RNG + 3 problems with compiling OpenCV in different configurations Vadim, Stonewall Ballard]
  • Started repairing the docs. The cxcore reference manual has been processed [Vadim]

Outlet Progress

  • Outlet detection: detection accuracy improved from 0% to ~25% on the latest dataset orange2x2_2009-10-30-12-53. The necessary step is gamma correction, the original images were taken with very high gamma and were over-saturated. Also, they differ from the datasets we had before by outlet scale and view angle. The current accuracy is still not high, but this is probably a matter of tuning the validation step following GHT. We will have an update of accuracy this week.
    One way descriptor has been optimized by including FLANN search over poses as opposed to exhaustive search. Even when we iterate through all nodes (exact solution) we get a significant speedup of 2-3×.
    • We should have automatic gama correction for more general purpose
    • One way descritor is not robust to a change in gama.
      • Ferns are robust to this but ferns like texture
        • Try LBP ? Would take some time
      • Different matching engine
  • Original FERNS code has been tested together with opencv implementation on outlet feature classification problem. Both even after parameter tuning have shown results significantly worse than one way descriptor. We believe that this is because of a lack of texture around each feature.
  • Geometric hashing method has been tested. A single edge from a training image was used for building a model. During recognition step, each pair of keypoint and edge that are close to each other are tested by finding best bases in the hash table and verifying them. The verification step just calculates the distance assuming an affine transformation given by bases, without recomputing it directly from point correspondences. The results are illustrated on images below. Blue shows the matched edges extracted from test images by canny, green shows the matched model. The algorithm is robust on orange outlets where orange blob borders are extracted well, but also works on white dataset where edges are available only partly. The method runs about 10-20ms per frame. In order to bring it in production it is necessary to improve the verification step that will filter out false alarms, and fuse it with feature classification engine.

Victor incoming 23rd

Code Coverage

Action Items

Gary

  • Send Victor link to Adaptive Regression Kernels
  • Talk with Blaise about gama other
  • Make sure Victor has building key
    • Get Victor an account

Vadim

  • Contact Brian to change the reporting format

Victor

  • Calandar descriptor: Have Alexi get this in removing Boost dependency, adding documentation and test code

From Last time

Gary

  • (./) Ask about how to upload SVN (Brian sent email, will do)
  • (./) Ask about how Bullseye handles dynamically loaded code — Brian answered(?)
  • (./) Get a place for Victor in the Green room — 6 new machines with reserved place for Victor
  • (./) Send Steve a note about contract renewal

Vadim

  • Pre-cluster the docs into language specific and all language
    • Then perhaps Anatoly can put in the latex conditionals
  • Make sure we migrate SVN and work with Brian to get test code coverage up and running

Victor

  • Send node about contract renewal
  • Have Alexey prepare Calandor feature for OpenCV inclusion together with
    • Documentation
    • Test code
    • Usage sample

2009-11-23

Agenda

  • Getting off of source forge
    • bugs
    • release instructions
    • SVN instructios
    • Messages to the user group
  • Coverage Progress
  • Documentation update
  • Grab cut
  • Calandor Descriptor
  • Handling TIFF images
  • NVidia
  • CVPR

Minutes

Login to OpenCV wiki

  • Just gave Vadim access

Getting off of source forge

  • Put links on opencv wiki directly to the download, Vadim to figure out and put on opencv wiki

Message to user group

  • When oepncv wiki is finalized, send another message to the user group

Coverage

  • cxcore – +2% (to 74%), cv – +2% (to 55%), highgui – +10% (to 32%).
  • 3 more big tests (for HOG, Face Detection and Video I/O) are in progress.
  • The current coverage chart is attached.

Documentation

  • Clustering is completed
    • Files are all universal instead of cxcore.tex we have cxcore_array cvxore_xml etc
      • cv.tex => cv_calibration.tex
        • These have big conditionals for python, C and C++
          • This is not fine grain conditional, but everything is all in one place now even if the documentation sentences are duplicated 2x in one document
          • Now a dedicated editor can feather in fine grain conditionals but maybe wait until python is switched to C++
      • highgui.tex stays
    • Next step is to fix how this appears online — do it this week
  • Attached is an example screen shot of the new pdf documentation

Background subtraction

  • In OpenCV
    • cvaux
      • Codebook
      • universal interface cv_bgStatModel
        • adaptive background mixture with shadow detection gaussian mixture
        • histogram per pixel
    • Use it to seed grab cut
    • GrabCut is done with kmeans with postprocessing for speed in OpenCV
      • Might be robust to outliers

New functions

  • GrabCut entered with test and sample, no doc yet
    • Document
  • Calandor function
    • Alexy Integrating Calander
    • Merge with planar patch detector
      • fern based and calaner are very similar
        • calander add prob dist, fern multiplied. Then use extremely random tree or fern based.
      • (Victor) Difference (calander extremely random and add, fern bayesian and multiply. Calander in ROS optimizes classification — not just probabilities but classification using random projection.
        • Suggest leave separate for now (choosing pairs of points is same, but the classification is different)
      • (Vadim) Thinks adaptation of calander to fern will be faster than keeping separate due to Boost lib dependencies.
        • Spent a bunch of time reshaping calander descriptor to be more like opencv style, interface and internal coding. Branch that didn’t use sse didn’t compile and had to re-write with the original code.
      • (Victor) Alexy is already studying Calander to remove Boost dependencies
        • Reshape code later (~2050)
        • Alexy is writing a sample for Calander
        • Should be checked against ROS output

C++ in PDF is reasonable form

  • TIFF image has tags
  • Adding alpha channel ~“photometric interpretation” RGB RGBA or something like mixed channel, write to TIFF
    • normally interleaved
    • OpenCV 3 to 4 channel, use mixChannels give 3 and 1 and copy to 4 channel

NVidia

  • Eugene at another company was talking about film demos (opencv running in heavy computer vision
    • Potentially a collaboration? They give data and reference code, we optimize
  • (Gary) Push them along
    • Edge blending example using OpenCV GPU to make seemless
      • 10Mpix rendered in real time using NVidia Cuda (Anatoly took 3 weeks, got 200x speed up)
  • Ideas
    • Do real time stereo on 10Mpix ExMor chips?
    • Block matching — has aperture problem, small has many false alarms even with high resolution due to image warping, large cannot catch fine details
      • Might limit the resolution of the disparity map
      • How big of an image can we do real time using Kolmogorov?
    • Real time high resolution Latent-SVM (Hog pyramid)
      • Just speeding up HOG

Look into octangonal integal images (SRI technology — they didn’t patent)

CVPR

  • workshops are closed
  • tutorials are Dec 4 (propose a tutorial)
  • Demos April 23 (robot based demo)

Vadim

  • 2 new tests have been added (by Anatoly Baksheev):
  • cv – grabcut
  • highgui – image I/O.
  • The code coverage was increased, respectively:
  • Documentation clustering has been completed. The produced PDF now consists of 3 parts: C API reference, C++ API and Python API. The table of content has been synchronized for all the 3 parts. The screenshot is attached.
  • Plans:
    • finish the in-progress tests,
    • implement test of stereo calibration & rectification
    • integrate calonder descriptor to OpenCV

    Outlet Detection

    Outlet detection: a new method for feature clustering based on geometric hashing has been implemented. This is an alternative to generalized hough transform that was shown to poorly perform for large viewangle (because larger viewangles cause larger object aspect ratios and this results in larger errors in calculating object center that is blurred between several bins). During the training phase GHash algorithm iterates through all bases built upon triplets of features detected on an outlet template and builds a hash consisting of bases referenced by corresponding affine coordinates of other features in the template. Online phase of GHash consists of iterating through a small subset of bases built on triplets of points from a test image, calculating votes for each of the bases from the hash entries and validating matches for bases with large number of votes. The method performs well on frontal images showing accuracy comparable to GHT, while being much better on orange2x2_2009-10-30-12-53 dataset (see the test results in the end of the message). The method works for 2×2 outlets, still has to be generalized for 2×1 outlets (this is a matter of making several parameters such as the minimum number of votes dependent on outlet template).

    A bug causing occasional crashes has been fixed in kdtree version of one way descriptor.

    Calonder descriptor port is in progress.

    Best Regards, Victor

    Test results with ghash:
    | Data | Results | Time |
    | cracked (correct/total): | 100/100 | Average time: 0.559697 |
    | fake_wall_2x2orange (correct/total):| 33/34 | Average time: 0.575073|
    | specular (correct/total): | 17/56 | Average time: 0.473343|
    | white2x2 (correct/total): | 99/104 | Average time: 0.597911|
    | orange2x2_2009-10-30-12-53 | (correct/total): 114/216 | Average time: 0.603217|

    Old Test results:
    | Data | Results | Time |
    | cracked (correct/total): | 100/100 | Average time: 0.392468|
    | fake_wall_2x2orange (correct/total): | 33/34 | Average time: 0.444598|
    | specular (correct/total):| 17/56 | Average time: 0.383454|
    | white2x2 (correct/total): | 99/104 | Average time: 0.381264|
    | orange2x2_2009-10-30-12-53 (correct/total): | 39/216 | Average time: 1.087916|

    Action Items

    Gary

    • Ask about allowing anonymous bug reporting in OpenCV Trac
      • ROS has mirrors around the world, what to do about that.
      • Put download links on OpenCV wiki?
    • Clarify team roles, make a case for 3&3
    • Check back with NVidia

    Vadim

    • Figure out how and where we should put SVN and releases and make appropriate links on http://opencv.willowgarage.com

    Victor

    From Last time

    Gary

    • (./) Send Victor link to Adaptive Regression Kernels
    • (./) Talk with Blaise about gama other
    • (./) Make sure Victor has building key
      • (./) Get Victor an account

    Vadim

    • (./) Contact Brian to change the reporting format

    Victor

    • Calonder descriptor: Have Alexi get this in removing Boost dependency, adding documentation and test code

    Alexey

    • Check that OpenCV Calonder output is the same as the ROS version

    2009-12-01

    Agenda

    • Code coverage and statistics
    • Documentation
      • progress
      • plans
    • Bug clearing
      • Open all bugs (even instantly solved ones, even if it is just a one sentence note) so that it can be closed and thus recorded.
    • Show case new function
    • External contribution history
    • More frequent SVN checkins, more by the rest of the team
    • Case for 3&3 team at least:
      • Need both to maintain/enhance the old code and advance the new code
      • New things that we’d like to get accomplished
        • 3D Model Capture
        • 3D rigid object tracking
        • Replace SIFT
          • Update Moped with this
          • Update 3D textured model capture
        • Visual odemetry
          • Image stitching
        • Scalable learning and feature architecture
        • Gabor jets
        • Latent SVM and extentions
        • State of the art face recognition
        • State of the art indoor people detection.
        • Transparent and shiny item recognition
        • Surface perception
        • New features such as steerable kernels
    • Plugging in progress

    Minutes

    Future plans

    • Need really usable tookits
      • Object category recognition toolkits
        • User has to write too much glue (Geometric hash, generalized HOG, selection …)
      • Geometric tool kit
        • Should be able to know camera pose
        • project
        • pose
    • Use as “drivers” next year
    • Object recognition pipeline
    • Tracking pipeline
    • Geometric toolkit

    Maintenance

    • Documentation
      • Clustering made
      • Online format is good now
      • Main thing now is to add and improve documentation
        • MSER
        • HOG
        • Chessboard ordering
        • Face detector
        • LBP (now part of cascade)
    • Need to expose interfaces for
      • LBP 9 sub-rectangle against central
      • HOG
    • Calonder descriptor
      • Testing it so that it re-produces original code’s results
        • Windows seems to have a bug since it diverges a lot
    • Bugs — what is outstanding, new and fixed
    • Make focus detection a “flag ship” contribution
      • What’s the best way to contribute code so that its usable without formal contibution?
      • Use several layers, use GIT
      • CMake is easy to add new files
      • Tests and docs is difficult
        • Need some unit test method such as pyunit
        • For documentation sphinx
    • For user contribution, copy ROS technique?
      • Have a project creation script that
        • Creates a directory
        • Puts in a CMake template
        • Creates a Latex file
          • Maybe an image directory
        • Creates a test directory
        • Creates a test stub
          • Create a data directory that is only pulled in if you build it?
        • Creates a sample directory
        • Creates a sample stub
        • Creates src and include directory
        • A simple blacklist file would stop compilation if it fails
          • OS dependent black list
    • Existing contributions

    Activities

    • several new tests have been added:
      • highgui (video I/O, GUI functionality) [Anatoly]
      • dense optical flow (is not fully complete yet) [Anatoly]
      • face & people detection [Maria]
      • KD-tree & FLANN [Maria]
      • kmeans [Maria]
      • calonder descriptor (not integrated yet) [Alexey]
    • These noticeably improved the function coverage, especially the highgui and cv.
    • online C++ documentation generation has been repaired, the table of content has been sync’ed with C & Python docs [Vadim]. Thanks to James, it’s now available online:
      • http://opencv.willowgarage.com/documentation/cpp/index.html
    • 2 bugs in highgui, 1 bug in cv and 1 bug in cxcore have been fixed [Vadim]

    PLANS:

    • add the following tests : ml’s EM and kNearest
      • stereo correspondence,
      • feature detectors and descriptors (SURF, Star, MSER, YAPE …) & planar object detection
      • background/foreground segmentation
    • add documentation for C++ cascade detector & hog detector.

    Plug and Outlet

    • Outlet detection: position of the robot forearm for plugging has been ratified by Stu (a compromise between perception and control).
      • Lots of experimental data have been collected.
      • There are almost no false positives in outlet detection (about 1 in several hundred images), but detection stability can be improved.
      • The algorithm works on geometric hashing that was adapted for 2×1 outlets. The experiments uncovered several bugs in geometric hashing that have been fixed.
    • The main obstacle for plugging in is high error in pose estimation (about 5-10mm, should go down to at least 2mm).
      • Error is in both outlet and plug pose estimations, and has both constant and random parts.
    • A small utility for saving images from a given topic has been implemented.
      • It is called extract_images and is a part of image_view package. It can be used to save images with a given frame rate from a bag file as well as directly from a robot.

    Action Items

    Gary

    • Create focus detector
    • Meet with Kurt about what he needs/wants
    • Work on “pipelines” schemes for next year as drivers

    Vadim

    • Send list of major contributions
    • Close out bugs, list what’s next, migrate to Trac
    • Run code coverage once more before Friday. Make sure machine learning is there

    Victor

    • Meet with Kurt about what he needs/wants

    From Last time

    Gary

    • Ask about allowing anonymous bug reporting in OpenCV Trac
      • ROS has mirrors around the world, what to do about that.
      • (./) (Vadim really) Put download links on OpenCV wiki?
    • (./) Clarify team roles, make a case for 3&3
    • (./) Check back with NVidia

    Vadim

    • (./) Figure out how and where we should put SVN and releases and make appropriate links on http://opencv.willowgarage.com

    Victor

    2009-12-02

    Agenda

    • Collection of mini-meetings
    • Discuss with Kurt what he’d like to see in OpenCV 2010
    • Discuss with James
    • Latest code coverage, some other statistics
    • Set up OpenCV twitter account

    Minutes

    OpenCV Twitter Account was set up

    • http://twitter.com/opencvlibrary

    Ideas for further opencv development:

    1. Bridge from opencv to eigen will be useful for people @ Willow (eigen is more convenient than lapack)
    2. A generic interface for descriptors, for instance, fast search for descriptors in a specific region
    3. Sparse bundle adjustment (needs [re]implementation?)
    4. Nister trees (available for submission to opencv)
    5. Point Cloud Library
    6. Stereo (Stereo Processing by Semi-Global Matching and Mutual Information Hirschmuller, improving correlation-based algorithm with projected light). Might want to switch to stereo correlation in opencv
    7. Turn opencv into ROS-like structure —
      • have independent “filters” that can exchange messages.
      • This will simplify programming — in some cases people will use a collection of these filters without actually writing and compiling code.
      • However this might be very platform-dependent.

    Future opencv pipelines to be attacked first:

    • Object recognition with pose estimation
    • Visual SLAM/Visual odometry (this is being rewritten for M3, is a candidate for inclusion into opencv with further development)
    • 3D image libraries

    List of Contributions

    General 2000-2009:

    • Linux and MacOSX makefiles (pre CMake-era)
    • swig-based Python
    • camera and video support in highgui Linux & MacOSX (video4linux 1/2, ffmpeg, gstreamer, xine, quicktime), GUI on MacOSX (Carbon)
    • OpenEXR and jpeg 2000 support

    2007:

    • SURF
    • O(1) median filter
    • trained haar-based cascades for fullbody detection, profile face detection
    • Octave bindings

    2008:

    • floating-point bilateral filter
    • unicap support in highgui
    • Google’s C++ image wrappers
    • Matlab-like sort
    • kdtree
    • at least 2 official patches (submitted via SF tracker) have been applied

    2009:

    • MSER
    • LSH (locally sensitive hash)
    • Spilltree
    • Self-similarity features
    • FAST corner detector
    • adaptive skin detector + fuzzy meanshift tracker
    • Entire, state of the art nearest neighbor library: FLANN
    • sparse bundle adjustment (Phase space)
    • faster boosted classifier prediction
    • Haar cascades for eye detection
    • Haar cascades for body parts and facial features detection
    • gstreamer writer + remote file support
    • draft version of OpenCV docs in RST format
    • at least 12 official patches (submitted via SF tracker) have been applied

    Code Coverage

    Action Items

    Gary

    • (./) Put coverage status into a spreadsheet googledoc

    Victor

    • (./) to talk to Kurt about status/improvements in stereo

    Vadim / *Victor*_

    • (./) to be included in the discussion on keypoints detectors /descriptors → have a better/unified interface for both in opencv

    2009-12-08

    Agenda

    • Twitter needs to tweet more
      • Every week, let’s say Tuesday, the “plans” should go there with names
      • Then people should say what they are working on every day, what they plan to do
        • Should I have an account on the opencv twitter?
    • A function is complete when:
      • It is coded (obviously)
      • It has documentation
      • It has good conditional test coverage
      • It has sample code, which might be the same as test coverage if it is well explained, but might be better if it is kept very simple.
        • A non-author should try to use the function from the documentation
        • For priority Willow code, we might want a “user test” here.
    • We need to collect a priority list of functions that should be at full compliance
      • I sent out a post to Willow
        • Functions Willow is using/wants to/plans to use (Calibration, Stereo, Homography, Affine, GrabCut, Calonder, FAST, background, things in support of visual odemetry … even Focus …)
      • Core image processing functions (arithmetic, logical, matrix ops, matrix reference counting,
      • Basic cv functions (Canny, Optical Flow, flood fill, convolution, edge detect, blur, threshold, morphology etc)
      • Basic learning: (Cascade, Main techique in FLANN, K-NN, k-means)
      • Others?
    • We will then make priority tiers and some basic time lines
    • All out focus this on the calibration tickets and documentation (about half the tickets might be documentation there?)
      • Vadim, take what resources you need including myself and Victor if we can be useful in this.
    • Adding circular element calibration patterns (What Victor is working on here)
    • Need everyone’s email … so I can remind them if they don’t twitter enough. I will create an internal willow opencv mailing list so that willow people can mail the team directly.
    • NVidia statement of work
    • Some comments about future strategy

    Minutes

    • People will tweet more often (1-2x/day at least) and tell them to use it informally … just what are you doing.
    • New functions should have a non-author use the documentation before the documentation is considered complete
    • Next release of code
      • _’May_’ sometime (for new functions and hardened code)
        • Put out intermediate release in _’Feb_’ sometime that has mostly fixes, docs, tests
    • Priority functions
      • Obvious simple ones such as
        • Image arithmetic, logic, creation/destruction
        • Cascade, Main techique in FLANN, K-NN, k-means
        • Canny, Optical Flow, flood fill, convolution, edge detect, blur, threshold, morphology etc
        • Willow ones: Calonder, one way, stereo, calibration …
        • Come up with candidate lists of priority functions ’by Monday
    • Calibration:
      • Victor to help with documentation
      • Change function to allow control of principle and focal point
      • Simple of way of getting min and max
        • 0: only good section, 1: all, ROI returned of entire and can pass values in between

    Calibration pattern with circular elements

    • Victor wants to change template matching in OpenCV in general
      • Feature detection and template matching as separate functions
    • Separate the topology mapper from the blob detector
      • Make this modular
      • Submit point matcher to OpenCV
      • Separate topology matcher
      • Generalized Hough transform
      • Geometric hashing
    • Make a very stable detector
      • Look at the SRI detector

    Draft a research plan going forward after Calibration

    NVidia

    • Can we get large speed up for HOG on GPU?
    • Stereo is a clear win on GPU however, that should probably be first
      • Correlation based
      • Stereo (Stereo Processing by Semi-Global Matching and Mutual Information Hirschmuller, improving correlation-based algorithm with projected light). Might want to switch to stereo correlation in opencv
    • Then put this together in an object segmentation/recognition framework to be inivestitgate

    Report

    • several new tests have been written, several old tests have been extended:
      new sparse optical flow tests, extended Farneback test [Anatoly]
      new SURF & StarDetector tests – the repeatability is checked [Anatoly]

    new tests for LSH (locally sensitive hash), PCA, reduce [Maria]
    2 new MLL tests: knearest, EM [Maria]

    new simple regression test of stereo calibration & rectification, based on samples/c/stereo_calib.cpp sample [Vadim]
    extended tests to cover C++ API: Canny, floodfill, Rodrigues, template matching, color conversion [Vadim]

    The current coverage chart is attached.

    • Fixed 6 bugs in the code and the documentation [Vadim]:
      trac ## 7, 8, 11.
      other bugs have been reported in opencvlibrary-devel or e-mailed to me (build on Win64, potential crash in SparseMat::convertTo, uninitialized memory read in bitwise_not).
    • Calonder descriptor is now in cvaux [Alexey]
    • Implemented draft version of the function cv::getOptimalNewCameraMatrix that should solve all the mentioned difficulties with OpenCV undistortion functions [Vadim]

    The spec is:

    
    Mat getOptimalNewCameraMatrix( const Mat& cameraMatrix, const Mat& distCoeffs,
    const Mat& R, Size imgSize,
    double minFScale, Size maxImgSize,
    Size& newImgSize);
    

    (A similar function will be added to C API as well)

    • The function takes the estimated camera matrix, distortion parameters and the optional rectification transform R, as well as the original image size. It computes the optimal new camera matrix and the new image size, so that the whole undistorted image fits the new image rectangle, or some part of it.
    • Some pictures from single-head camera are attached.
      • wide_angle_calibration_orig.jpg – original frame from wide-angle camera (thanks to Victor for the image and the associated camera data)
      • undistort_small.jpg – the image resolution is kept the same, and the new focal length is computed so that the image fits entirely.
      • undistort_large.jpg – the image resolution is increased by 20% at max, while the new focal lengths fx, fy are chosen close to the original ones.
      • any other reasonable variant is possible with the function too (e.g. keep the focal length intact, make the image as big as necessary to fit the undistortion result).
    • One stereo camera the function is being debugged.

    PLANS:

    • continue to increase cxcore & cv coverage.
    • add *James*’ Python tests into the regression testing loop
    • close urgent other tickets at Trac and SF bug tracker.

    Outlets

    • Outlet detection:
      • pose estimation for outlet detection has been stabilized. The distance between the tip of the plug and the ground hole that the plug touches is within a range of 0.1-3mm.
      • Outlet and plug nodes have been changed from working with prosilica poll service to forearm messages processing.
    • Texas parking:
      • A color template detection method has been implemented for texas parking. It is based on the code that was used for outlet detection (hole detection + geometric hashing).
      • A video from texas camera has been obtained by hacking a gstreamer plugin.
      • Now working on a ros node that will actually drive texas to the parking.
    • Other: A lot of talking to opencv stakeholers, trying to boost the communication between opencv and willow teams.

    Action Items

    Gary

    Vadim

    Victor

    From Last time

    Gary

    • <:( Create focus detector
    • (./) Meet with Kurt about what he needs/wants
    • :\ Work on “pipelines” schemes for next year as drivers

    Vadim

    • (./) Send list of major contributions
    • Close out bugs, list what’s next, migrate to Trac
    • (./) Run code coverage once more before Friday. Make sure machine learning is there

    Victor

    • (./) Meet with Kurt about what he needs/wants

    2009-12-15

    Agenda

    • Coverage statistics
      • We must generate a list of functions to be covered for next release
        • One algorithm for this is: All code used by Willow + any prominent bugs from the community
        • All new functions need: Conditional test coverage, docs, sample code.
      • The calibration pipeline (mono, dual, rectification, alignment, correspondence, disparity and depth) is complex and
        • may need its own tutorial. I will try to help write some of this …
          • but where to put this — enlarged section of the manual?
        • Needs a more comprehensive testing plan
        • Calibration also should have more general pattern finders: Element matchers separate from their geometry … at least on the plane
    • Twittering — need to do more often, it’s just a snapshot of the state.
    • OpenCV expansion
      • NVidia statement of work is out
      • Possible to get Intel involvement for some matrix friendly operations that they want
    • Future research work (builds somewhat on the calibration genreralization)
      • In talks with Radu and myself, will be tied closer to Willow deliverables
      • Rigid, textured item recognition
        • Has connections with visual odometry

    Minutes

    Coverage

    • According to literature, 100% conditional coverage means that the actual code conditions are about 50% covered
      • Not worth going to 100%, more like static in 90%s and Conditional > 70%
        • Especially, we’re not changing the API anymore, so …
    • Need to make a list of priority functions
      • Functions that Willow uses
      • Functions with bugs
      • Chart this also
    • Bug reports on community wiki have to be re-directed to Trac page

    Calibration

    • Needs sort of a tutorial
      • Build around Stereo Calib sample
      • Cases in Calibration
        • Very good lens
        • Birds eye
        • One shot
        • Other calibration advice
        • Maybe look through Jean-Yves Matlab tool box and take some of the same examples
        • Examples of varying how much of the image is in or out of the rectified
    • High distortion images need looking out
      • Kurt/SRI had a square matcher that was more robust but slower … as an option?

    Research plan 2010

    • Geometric arrangement of interest points
      • Moped library kind of does this, but with SIFT and licensing problems
      • We also want to handle: Mono, Stereo and deformable objects/cloth
      • Look for low resolution detector/descriptors (?)
        • Daisy
        • High res … !LePetit
        • Calonder descriptor
          • The code is already in cvaux library. We are preparing some test code for it. We probably need to unify some of the framework (ferns, oneway,…) and then have a detectors/descriptors overall framework. We also want to add deformable region detectors.
          • Its in cvaux.hpp (in includes) Look for RTreeClassifier at the bottom.
          • It needs documentation for to be moved into cv.
    • 10MPix stereo

    Google Summer of Code

    • Focus around collage, stitching

    Reports

    Vadim

    • OpenCV twitter is now being updated regularly (http://www.twitter.com/opencvlibrary) [all]
    • stereo rectification & undistortion improved, extended cvStereoRectify, added cvGetOptimalNewCameraMatrix;
      • it’s now possible to control the visible area after undistort/rectification;
      • different branches of the “camera calibration” section in the ref.manual have been merged and the content was improved [Vadim]
    • several new tests have been written/improved:
      - Calonder descriptor – new test [Alexey]
      - estimateRigidTransform – new test [Anatoly]
      - 3D affine rigid transform – new test [Anatoly]
      - SURF & StarDetector – improved robustness of the test [Anatoly]
      - big new regression test for elementary matrix operations (cxmat.hpp) [Anatoly]
      - Farneback’s test – more branches tested [Anatoly]
      - kdtree & flann – more branches tested [Maria]
      - haar & lbp object detection – more branches tested [Maria]
    • A LOT of bugs have been fixed:
      - trac: tickets #10, #12, #13, #29, #34, #36 [Vadim]
      - SF: #2906973, #2909108 [Vadim]
      - also insufficient accuracy in XML/YML in the case of some floating-point numbers [Vadim]
      - 3 bugs in MLL and traincascade, reported by JIA PEI [Maria]
    • We now use *James*’ unit tests in Python in our regular testing James, Anatoly].
    • The current coverage chart is attached below

    PLANS:

    • continue to increase cxcore & cv coverage.
    • close remaining major tickets in trac and SF
    • extend cxcore & cv documentation.

    Victor

    • Texas is parking fairly reliably, at least in its current location
    • Software is committed to sandbox
    • Web button is made by Josh for Texas behavior

    Pluggin in

    • We have reliable hole and chessboard detection from *Victor*’s code
      • Except for a sensitivity to light/gamma correction setting.
    • Full development notes in PluggingIn

    Action Items

    Gary

    • (./) Re direct bug reports in wiki to the Trac page
      • (./) Send out message to group about this

    Vadim

    • Work on developing a test set list
    • Docs and tests
    • (new) We need to consider a move to Eigen
      • It’s what Willow has standardized on
      • Radu used this and beat SSE optimization by Kurt and Ethan. Eigen is well thought out in this way.

    Victor

    • Research direction: “Textures Geometrically Arranged” or “TAG” toolbox
      • We need to study Radu’s PCL (Point Cloud Library) and develop a compatible architecture for the geo-feature matching
        • For one, we need to have a nice framework to:
          • Find interest points (or do dense or sub-sampled-dense)
          • Compute descriptors there
        • We need to look at !LePetit and Fua’s work since we will also want to handle deformable later

    From Last time

    Gary

    Vadim

    Victor

    2009-12-22

    Agenda

    • Use of IPP in general and latest AVX extensions for IPP in particular
    • Plan of record
    • Geometery and Texture
    • Eigen
    • Docking and plugging in

    Minutes

    • IPP
      • No latest extensions
      • Have to choose IPP at configure stage
      • They should sponsor someone for their new chip and IPP since we’re flat out working on Willow and community issues.
    • Plan of record until Jan 15: https://docs.google.com/a/willowgarage.com/Doc?docid=0AZyRqEyzcAajZGZmanozNTZfMTJuZzR3em1keg&hl=en
      • Make spreadsheet that tracks that
      • Along with general tracking
    • Geometry and Texture
      • For now, Victor is
        • looking into speeding up the chessboard finder when there is no chessboard in the image.
        • Also stabilizing the locations of the corners for sub-pixel accuracy (creates noticable differences in calibration results)
        • Looking into separating features and detectors … probably do as a separate function rather than trying to re-write the existing calibration.
      • Have SIFT as a comparison
      • Need unified interface with fast search
      • Also edges to consider (Zisserman’s local shape descriptor) combined with affine invariant edge trackers
      • Also deformable and flexible objects (look at !LePetit and Fua’s work).
    • Eigen, ROS has standardized around this, need to work with it seemlessly.
      • Vadim has been considering, had a good idea of how to implement
        • Not so difficult to implement hopefully (passing Eigen matrices to any OpenCV functions).
        • Idea: Not to switch completely (drop OpenCV for Eigen) because OpenCV is more flexible.
          • I/O with Eigen should be seemless
    • Docking
      • “Too much CPU”, but it doesn’t have to run all the time
      • No feedback … can improve in many ways, need to know if someone is working on.
      • ROS node is not up and running.
        • ssh to Texas and run launch file and press the button. It will park
          • Node should be a service

    Action Items

    Gary

    • Send Victor sample sequences of textured objects under known pan and tilt
    • Find out who can be the “consumer” to get docking working better.
      • Talk with Josh Tyler

    Vadim

    • Add calibration functions as extra line to spreadsheet

    Victor

    • Speed up chessboard finder when no chessboard is in the scene
    • Stabilize sub-pixel corners
    • Find SIFT code that we had before so that we can use it for a comparison study

    From Last time

    Gary

    • (./) Re direct bug reports in wiki to the Trac page
      • (./) Send out message to group about this

    Vadim

    • (./) Work on developing a test set list
    • Docs and tests
    • (./) We need to consider a move to Eigen
      • (./) It’s what Willow has standardized on
      • (./) Radu used this and beat SSE optimization by Kurt and Ethan. Eigen is well thought out in this way.

    Victor

    • (./) Research direction: “Textures Geometrically Arranged” or “TAG” toolbox
      • We need to study Radu’s PCL (Point Cloud Library) and develop a compatible architecture for the geo-feature matching
        • For one, we need to have a nice framework to:
          • Find interest points (or do dense or sub-sampled-dense)
          • Compute descriptors there
        • We need to look at !LePetit and Fua’s work since we will also want to handle deformable later

    2009-12-30

    Agenda

    • Progress update on Plan of Record.
      • Thanks Vadim for adding this category to the spreadsheet.
    • I like the twittering, more is better.
      • Not accomplishments, anything you are currently doing: “Writing text for X, bored and has a bug, think I’ll play some tetras for awhile”.
    • Calibration progress
      • Chessboard rendering
    • Geometery and Texture, would like to start this
      • Separate detectors/descriptors and their geometry
      • Edges, especially shape edges
      • Flexible objects
      • Nister trees
    • Plugging in (Probably nothing much until Jan).
    • Texas (Unknown status right now, could try to gather more data).

    Minutes

    • Plan of record
    • Spreadsheet of calibration progress.
    • All on vacation until the 11th. Victor 3rd-7th, Jan 23?
    • cv:
      • Proposal to use move some seldom used functionality into cvaux
      • Move experimental functionality (new stuff) into cvcontrib
    • Segmentation GrabPoints
      • Soft points needed — they are there, need to be exposed
    • Calibration finish point. Holidays push off 100% coverage point
      • Improving accuracy — subpixel [Victor]
      • Robustness of detection. No known examples right now
      • Calculate the errors of calibration such as Jean-Yves does by Jan 12? [Victor]
      • Want automatic chessboard rendering as a sample
        • [Vadim]: I asked Anatoly to prepare the sample (actually, to clone and modify calibration.cpp sample) that uses the chessboard generator.
    • Texture and Geometry
      • Need architecture, Victor propose, look at Radu, get Patrick involved
      • Want to extend to flexible objects
      • Nister tree
        • ROS one tied to Calonder descriptors
        • Want it to be flexible
    • Plugging in wait until Jan
    • Texas —
      • click and go there
      • improve template recognition
    • !GraphCut based segmentation with fixation (similar to !GrabCut, but uses log-polar transformed space)
    • GSOC
      • Port OpenCV to Android
      • OpenCV in a cloud (Gphone image goes into a server and processing happens across a cloud)
      • Image stitching
      • Recognition of ground plane, place for augmented reality.
      • Texture based object recognition — ease of use on the phone
        • Flexible objects

    NEXT MEETING: JAN 12, 2010

    Reports

    Victor

    • Chessboard detector: a function сvCheckChessboard(…) that runs a fast test on whether there is a chessboard with a given number of corners in an input image.
      • There is a new flag in cvFindChessboardCorners(…), {{{CV_CALIB_CB_FAST_CHECK}}},
        • if it is on, then this check is performed before the main algorithm and if a chessboard is not found, the function returns 0 instead of wasting 0.3-1s on doing the full search.
        • The check works correctly on all test images we have in the algorithmic test.
      • A new algorithmic/timing test has been implemented for сvCheckChessboard(…).
      • The running time for cvFindChessboard(…) in the case when there is no chessboard in the input image has dropped ’from 0.3-1s to 0.01-0.05 s per frame’.
        • This fixes the opencv bug report https://code.ros.org/trac/opencv/ticket/41#comment:3.
    • An algorithmic test on cvFindCornerSubpix(…) that uses artificial chessboard generator is on the way.

    Vadim

    • This week was mainly spent in improving camera calibration functionality coverage:
      • functions coverage: 65→76%
      • conditional coverage: 64→67%
    • Here are the details:
      • 7 new tests have been written,
      • 4 existing tests have been extended:
        • created 2 huge tests of various inline operations on matrices (matrix expressions) and other data structures [Anatoly].
          • 370 out of 750 such functions are covered now.
          • 11 bugs have been found and reported
      • implemented artificial chessboard generator for the new set of strict calibration tests [Anatoly]
      • extended PCA test, 1 bug found [Maria]
      • implemented regression test of drawing functions [Maria]
      • implemented test for the new C++ calibration functions, cvCalibrationMatrixValues and cvStereoRectifyUncalibrated [Maria]
      • implemented test for cvUndistortPoints, extended the test of cvUndistort [Alexey]
      • extended cvGEMM, cvTransform tests [Vadim]
      • 10 bugs have been fixed (input data corruption in cv::eigen, Trac ## 20, 35, 38, 39 (several small bugs), 43, 44, 46, 51, SF #2907097) Vadim, Maria]
      • OpenCV test engine was improved [Vadim]:
        • now the detailed information about each failed test is printed right into the console
        • “-r” mode was added, in which the engine keeps running even if some of the tests crashed.
      • ~30% of cv functions using CV_ERROR, BEGIN, END_, CVCALL etc.
        • (the old-style error processing) have been modified to use c++ exceptions (CV_Error(), CV_Assert()).
        • The subset includes all the calibration functionality, except for the chessboard finder (since it has been modified by Victor at the same time).
        • That is, now we can implement “bad arg” tests for the calibration functions.
    • The updated code coverage statistics is here
    • Vacation: it looks like there will be official holidays in Russia all-the-way to 11th of Jan (updated information will be sent tomorrow), so

    Happy New Year to everyone! We are going to holidays vacation.

    Action Items

    Gary

    • Edit C++ interface tutorial
    • (./) Send out log polar “grabcut” paper

    Vadim

    Victor

    • Send texture and geometry API around
    • Subpixel accuracy improvements for calibration.
    • Adding calibration error estimates

    From Last time

    Gary

    • :\ Send Victor sample sequences of textured objects under known pan and tilt
      • I’m coding up the pan tilt stage control this week
    • (./) Find out who can be the “consumer” to get docking working better. (Melonee)
      • (./) Talk with Josh Tyler

    Vadim

    • (./) Add calibration functions as extra line to spreadsheet

    Victor

    • (./) Speed up chessboard finder when no chessboard is in the scene
    • (./) Stabilize sub-pixel corners
    • {i} Find SIFT code that we had before so that we can use it for a comparison study