OE-4 "OpenCV 4.0" discussion

[vpisarev]

the feature request about evolution proposal OE-4

[kinchungwong]

Here is a comment to users who might object to OpenCV's requirement of C++11.

(Disclaimer: This comment is submitted by a member of public, therefore it does not reflect the position of OpenCV team unless this comment is adopted.)

Perhaps these users can consider Google Abseil C++.

Because Abseil uses Apache License version 2.0, it is not the same as OpenCV BSD license. For this reason, it is not to be packaged in source-code-form together with OpenCV.

At best, user (who must use an older compiler) need to download Abseil separately, and then point OpenCV CMake to that location, and the CMake will contain configs to facilitate the use of Abseil during build.

This suggestion, if adopted, does affect how OpenCV source code could be allowed to use C++11/14/17. It requires a different namespace. This means OpenCV cannot directly use C++11/14/17 types from namespace std, instead it must use a different namespace (to be determined, an example might be namespace cvstd), which allows CMake to redirect to namespace absl if the user specifies the Abseil option in CMake.

The feasibility of this suggestion hasn't been thoroughly researched, so there may be additional roadblocks. If project resource permits, OpenCV might want to invite comments from people who are knowledgeable about Abseil C++ integrations.

[LaurentBerger]

Add training method in DNN module (caffe)

[catree]

@LaurentBerger
In my opinion, it is not desirable to have a DNN training module comparable with Caffe in OpenCV:

• there are already many deep learning libraries (Caffe, Caffe2, Tensorflow, MXNet, CNTK, Torch, ...)

• the field is moving really fast, to be useful the module needs a strong support and adoption from the open source and scientific community

• if it is just to support conventional DNN network training, why not rely on well-established DNN library?

• DNN training takes time, currently CUDA seems the only working solution for efficient DNN GPU training?

• it will take lots of effort to develop and maintain a module like this in my opinion

I would rather prefer that a standardized DNN model like ONNX (or DNN OpenVX?) be adopted by the DL community to allows network training in the common DL libraries and easily import the model in OpenCV for inference.

[kinchungwong]

@LaurentBerger @catree

Pardon for inserting myself into a discussion for which I am not qualified. I have never used any of these tools before. (When I got my MSc, ANN wasn't as hot as it is today, and none of today's tools existed back then.) However, I would like to offer perspective from the point of software integration.

Let's treat each integration scenario separately:

• Scenario 1: training
• Scenario 2: inference
• Scenario 3: user's software stack is C++
• Scenario 4: user's software stack is Python

In all integration scenarios, there needs to be some loose-coupling, or product management controls that prevent interface-breaking and behavior-breaking changes from causing ripples that destabilize the entire system. If designed well, these loose-coupling techniques shouldn't adversely impact user experience. It will definitely increase the library development effort, the complexity of build systems, etc., but all of these would "seem automatic" from user's perspective. The library's maintenance team will absorb the cost.

There are so many things OpenCV wants to integrate with, so a plugin architecture is desired.

We might realize that integration in Python is much easier. Also, most users use Python to do training. If the ("training + Python") part of story is handled well, then OpenCV has partly satisfied the user demand.

"Inference + C++" is the more difficult part, because high-performance (low resource consumption per unit of useful work done) and dependability (as used in unattended software systems) become important. I'll let others who are actually familiar with these libraries to speak.

[vpisarev]

On bringing the training to OpenCV DNN - we are not going to do it, in the nearest future at least. We'd better to rely on 3rd-party software. At the same time, we will continue to develop various scripts, mainly for TensorFlow, to post-process/optimize/quantize trained networks: https://github.com/opencv/opencv/tree/master/modules/dnn/misc

[vpisarev]

@kinchungwong, I think, I did not quite understand your comment. Can you please point me to the original comment where people object to our decision to move to C++ 11.

It's not only about the standard library, of course, it's also about the language features that we are going to use, such as lambda functions, initializer lists etc. Also, some newly developed (not integrated yet) modules of OpenCV are written in C++ 11, so in order to integrate them, we will have to switch to C++ 11.

[kinchungwong]

@vpisarev I apologize. When I "hypothesized" that there might be people who might object to moving to C++11, I was thinking about users who are still on OpenCV 2.4.x. There was a jump in my logic in which I thought their dependence on older compilers might be the root reason that they had to stay with both an older version of OpenCV, as well as older version of C++ compiler and language features.

I agree that new code are typically written with new language features, and some of those new language features (lambda, initializer lists, etc) cannot be emulated with libraries or macros. OpenCV depends on collaborations and contributions, which means OpenCV depends on acceptance of C++11 code. Meanwhile, OpenCV users depend on OpenCV, so they indirectly depend on getting enhancements and bugfixes which contain C++11 code too.

Perhaps OpenCV can set up a place (a page on Wiki, or a sub-site under OpenCV Answers for example) where, if anybody has questions about OpenCV's move to C++11, they can ask there. Similarly for users or projects which continue to depend on OpenCV's public C API.

[vpisarev]

@kinchungwong, ok, I see.

Regarding the place where one can ask some questions - I think, we have such a place :)

[hrnr]

Couldn't we move straight to C++14? C++14 is basically bugfix of C++11, but has been really useful for my projects (I use mainly templated lambdas a lot + better constexpr). I remember a few places in OpenCV where better constexpr would help with precomputed stuff. Which compilers/dependencies are holding us back?

[DavideCatto]

It is desirable to have some of the unsupported layers into DNN module like Rsqrt, Maximum or simple l2_normalize very used for embedding networks, facenet and so on.
Thanks for your works.

[dkurt]

@DavideCatto, there is NormalizeBBoxLayer which actually does Lp-normalization. Other simple layers could be implemented using custom layers support that is going to be merged from a PR #11129.

[vpisarev]

@hrnr, C++ 14 is not supported well until GCC 5.x and is made default only in GCC 6.x. The still relevant MSVC 2015 does not support many C++ 14 features either (not to mention that many people still use MSVC 2013, which does not support all of C++ 11). This is the reason number one. The reason number two is that for OpenCV, the big library which needs to be supported for years, we do not want to use all the coolest features that C++ has; we want to keep the code simple and maintainable. So, let's be more conservative.

[hrnr]

If this is the situation with MSVC, I understand that decision. Thanks for reply.

[petersupan]

A Feature which I consider really important for OpenCV 4.0 is an increase of the maximum cv::Mat area to more than 4gb/int32 (the limit caused problems for me several times).
It is explained in this discussion #10211.

Since Binary compatibility will be broken in this release anyway, it's a good chance to tackle the problem.

[kinchungwong]

For the large matrix support mentioned by @petersupan , namely increasing the maximum supported cv::Mat area to more than pow(2,32), can this item be made into its own Evolution Proposal item, so that there is a dedicated thread to discuss the design, mechanics, implementation details, and task list? The "task list" would refer to the ripple of code updates needed elsewhere (e.g. image processing code), in order to utilize such large matrices as input and output. It is also good to have a task list for new unit tests that needs to be written.

[vpisarev]

no need to do radical changes in this case; we actually tried to fix it a few years ago already; #11505 seem to complete this effort.

[kmhofmann]

Not moving to C++14 (and updating the smelly parts of the codebase a bit in the process) seems like a very short-sighted decision. C++14 is pretty established, given that it's 2018 by now. Many big projects are already using it. (If Chromium can do it, so can OpenCV.) Even C++17 is pretty much ready to be used in production scenarios.

The argument that poor souls with older compilers will need to be supported doesn't really hold, since:

• they will upgrade sooner or later, while OpenCV will be stuck with a bad decision for an unspecified amount of time, and
• there's always still OpenCV 3.x for this small group; that doesn't vanish overnight.

And if you want to make(!) the code simple and maintainable, then C++14 is a good step over C++11. Mainly due to multiple "bugfix" features, but also extended constexpr support.

[asmorkalov]

@vpisarev I think it's time to close the OE.

[kmhofmann]

Interesting to see my comment from 2018 again, which is still very apt (and apparently liked). Switching to using only C++11 in 2018 was far too conservative. In 2020, it's overdue to move to C++17 in preparation for supporting usage of C++20 features in about a year or so. I guess that will come for OpenCV in 2032. Sarcasm aside, this is about the possibility of making code simpler, safer, and easier to read and maintain, and there is a chasm of a difference between C++11 and 20. And OpenCV has a lot of deficits in this regard...

[asmorkalov]

@kmhofmann OpenCV team works on the scope for OpenCV 5.0 release and we are discussing the standard update each time. The conservatism is not just somebody's personal decision it's matter of fact that is OpenCV used on a lot of platforms with different compilers and we need to support bindings to other languages.

[kmhofmann]

@asmorkalov These are the usual arguments, but I don't think they should be valid. Again, people will (and should!) upgrade to newer compilers sooner than later. They actually could be encouraged by projects like OpenCV and move the ecosystem forward instead of holding it back, which is what OpenCV is implicitly doing. And, very obviously, old versions of OpenCV don't just vanish into thin air but can still be used by people that are unwilling to move with the times -- until they are, or have to.

[alalek]

holding it back

OpenCV doesn't block upgrading on newer C++ in your applications (in general you don't need to recompile OpenCV library, however you can do that).

[PhilLab]

The switch to a newer language standard just defines which language concepts can be used to improve OpenCV itself As alalek said, your code is independent of the library's standard.

And looking at MSVC, their support for even C++17 is still poor: https://docs.microsoft.com/de-de/cpp/overview/visual-cpp-language-conformance?view=vs-2019

So having the newest standard for OpenCV would mean that programmers using a powerful compiler would commit OpenCV patches which break builds of less powerful but widely-spread compilers. You would have any right to complain and laugh about MSVC, but the fact is that it is widely used and thus, OpenCV must compile under this platform.

And this is probably just one example of many. As a conclusion, a popular library like OpenCV would cause many problems when always switching to the latest standard. Unfortunately, they have to adhere to the speed of the popular (and also some less popular) compilers.

To be honest, I am very excited that C++11 is on the table :-)

[kmhofmann]

just defines which language concepts can be used to improve OpenCV itself

Of course, I know. And that's exactly what I meant, @PhilLab , @alalek . Not adopting newer versions of the C++ standard as a project is holding back an ecosystem in the big picture, including people who want to learn and get inspiration from what the "big" projects do. (Not that I recommend anyone taking inspiration from OpenCV, tbh...). By adopting newer standards more aggressively, users are effectively pushed to upgrade their toolchains, something they should have been doing in the first place. The net effect is only positive.

And looking at MSVC, their support for even C++17 is still poor

I don't think this is actually the case, or you're reading the table wrong. Here is a better organized overview that C++17 support of MSVC is actually very good, minus some quite minor features that could be disallowed: https://en.cppreference.com/w/Template:cpp/compiler_support/17

programmers using a powerful compiler would commit OpenCV patches which break builds of less powerful but widely-spread compilers

This is a completely hypothetical issue (read: not an issue in practice) which can fully be caught by good CI/CD.

To be honest, I am very excited that C++11 is on the table :-)

I am more pessimistic here; the adoption was about six years too late.