Add OTB provider to qgis processing plugin [processing]

[rkanavath]

Description

This adds otb provider to qgis processing plugin. Here are some highlights.

• OTB descriptor files are not included/maintained in qgis source tree.
• OTB provider is generic and works for all OTB version starting from 6.6.0
• Testing of OTB algorithms has been added (using .yaml and others).
• No changes into qgis core other than a python import in Processing.py.
• Travis ci script (linux) is added to install otb package. (osx will be added if linux ci pipeline passes)

Checklist

• Commit messages are descriptive and explain the rationale for changes
• Commits which fix bugs include fixes #11111 in the commit message next to the description
• Commits which add new features are tagged with [FEATURE] in the commit message
• Commits which change the UI or existing user workflows are tagged with [needs-docs] in the commit message and contain sufficient information in the commit message to be documented
• I have read the QGIS Coding Standards and this PR complies with them
• This PR passes all existing unit tests (test results will be reported by travis-ci after opening this PR)
• New unit tests have been added for core changes
• I have run the scripts/prepare-commit.sh script before each commit

[rkanavath]

@luipir , could you have a look at changes. travis tests are fixed now

[luipir]

I'll give a try... but I'm busy on many front (userconf) in this moment. No problem if someone else is able to review before me

[nyalldawson]

I'd like to review before merge too, thanks!

But upfront: Thanks @rkanavath --- looks like great work from a preliminary read over 🎉

[luipir]

But upfront: Thanks @rkanavath --- looks like great work from a preliminary read over

of course, thanks @rkanavath :)

[rkanavath]

@nyalldawson , given that last commits are okay is it possible to move this to 3.6 milestone?

[pcav]

so the plugin is not trying to install, the use can do it in his usual way. thanks.

[rkanavath]

@pcav , yes plugin know nothing of how or if otb is installed. The change in travis docker is for testing in travis-ci.

[nyalldawson]

This looks good to me now. There's going to be some issues with interoperability with other algorithms in models due to the layer handling (i.e. not supporting memory layers), but that can always be addressed later and further refined.

@alexbruy did you want to review too?

There's still the outstanding question about the docker image changes too, which either @3nids or @elpaso will need to review

[SrNetoChan]

Wasn't it decided that except for GRASS and SAGA all the other providers would stay outside the Core, installed as plugins?

[rkanavath]

Thanks @nyalldawson and others for your help during review process. I really appreciate your inputs. Plus lot of great changes in processing which made code for provider much simpler!

I agree with you that in memory provider support can be added later. I will add this as a todo in otb bug tracker.

Regarding docker image changes, I don't have an alternative for testing otb. So will wait for inputs from @elpaso or @3nids.

[alexbruy]

@nyalldawson, sorry I won't review it. My position did not changed, this should be a 3rd party plugin like all other Processing providers. I don't see any reasons for exceptions, we had agreements "no more python plugins in the core" and "move processing providers outside core".

[pcav]

@alexbruy I can't find it back right now, but we have been discussing this case rather extensively, and the final decision was to include OTB

[alexbruy]

@pcav I perfectly remember what and how was decided

[nyalldawson]

@alexbruy

That's fine -- I totally get where you are coming from. If there's no objection to the docker changes I'll merge after thaw.

[rkanavath]

@nyalldawson, @pcav is it possible for someone else to review changes to docker file in travis to complete it. If @elpaso (I tried to contact him by mail) and @3nids too are not agreeing (don't want to understand why is that) to include otb provider and thereby refusing to even review changes what will be remaning option left.?

Even though docker changes aren't essential for functioning of provider itself but it allows for easy testing and helps improve integration and communication between two projects.

Thanks in advance.

[pcav]

I agree. Unfortunately, I don't know the docker infrastructure, so I cannot be of direct help here.

[nyalldawson]

It's ok, it's tagged as "merge after thaw", so we are just waiting on feature freeze to lift and I'll merge

[pcav]

Great news, I'll test soon after merge.
Thanks.

[vpoughon]

Hi all,

It's great to see that this PR is going to be merged. Thanks for all the work!

I'm a developer and PSC member of the OTB project, just a quick note: Reading this thread I see that there is a bit of friction about where this code belongs. IIRC, the main reason we pushed for the OTB provider to be part of QGIS core is that we had a few issues with API breakage. When it was outside of core the plugin suddenly stopped working for users who upgraded their QGIS version. If you can find another reference to a discution about this please let me know.

On our side, integration with QGIS is an important feature. We see a lot of users relying on both of these osgeo projects and we just want what's best for them. For this we are willing to spend ressources and work on QGIS integration in the years to come (i.e. assign dev time to make QGIS PRs, or whatever else is the best technical solution).

There's also the issue of documentation, and installation of the plugin. Both of these are a complete mess from a UX perspective given the state of QGIS/OTB/plugin version compatibility matrix.

Ultimately I feel like the question is about who should maintain this code. I'm not well versed in the technical details of this (@rkanavath did all the work), but I just want to make clear that we are not dumping this code in the QGIS repo hoping that it will just keep working. We are open for any discussion necessary (both technical and political) about how to make the best OTB-QGIS plugin for users. We have a gitlab issue tracker and a new discourse forum, both with github SSO, or feel free to reply here.

Thanks a lot

[pcav]

Hi Victor,
thanks for stepping in. I assume that if you take responsibility for this area of code, there is no reason not to include it. I fully support your efforts.

[pcav]

Compiled and tested: it requires to add manually the paths to OTB folder and OTB application folder. Tried /usr nd /usr/bin, but I still get errors.

[pcav]

No OTB algorithms found in '/usr/bin'. OTB will be disabled

[pcav]

ls /usr/bin/otbcli*
/usr/bin/otbcli /usr/bin/otbcli_MorphologicalClassification
/usr/bin/otbcli_BandMath /usr/bin/otbcli_MorphologicalMultiScaleDecomposition
/usr/bin/otbcli_BinaryMorphologicalOperation /usr/bin/otbcli_MorphologicalProfilesAnalysis
/usr/bin/otbcli_BlockMatching /usr/bin/otbcli_MultiImageSamplingRate
...

[rkanavath]

@nyalldawson Thanks for merge!.

@pcav can you check this readme for configure otb.
https://gitlab.orfeo-toolbox.org/orfeotoolbox/qgis-otb-plugin#open-processing-settings
If you have stil issues, please report on https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb
and we can discuss it there.

[pcav]

Thanks @rkanavath for the hint, now it works with: /usr/lib/x86_64-linux-gnu/otb/applications/ and /usr. Could you please add these standard paths to the man page, rather than your custom paths?
More generally: why is this necessary? Other backends do not need a config, and this is way easier for the user.

[pcav]

Tested with Segmentation, all default params, it fails because of a missing compulsory Condition parameter, which I cannot find in the interface.

[pcav]

Large scale meanshift fails with raster.out.tif files are not supported as outputs for this algorithm.
Should I report these as tickets on https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb ?

[rkanavath]

Should I report these as tickets on https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb ?

yes, please. More otb developers can look into this if on gitlab. Thanks a lot for your effort in testing. I very much appreciate it. We will work on it and fix remaining issues in otb.

[alexbruy]

This is core stuff now, why use 3rd party bugtracker? IMHO all issues related to the core should be reported in the QGIS bugtracker. This will give us statistics and won't confuse users

[pcav]

This makes sense to me, thanks @alexbruy
If the issue is upstream, it can be moved by the developer.
@rkanavath please confirm this is the way to go and I'll open tickets.
Thanks.

[rkanavath]

@alexbruy , I was refering to issues related to particular algorithms/ and maybe specific version of OTB. The issue is not about provider code or all algorithms or qgis core part. If no algorithms are working I agree that it should be discussed on qgis. It all depends on which part of the code is having problems.

@pcav, you can report to qgis bug tracker and we can move it to otb tracker if required. Anyway discussion should be kept out of a closed pr. Sorry that I was still new to workflow in qgis. But I am learning a lot .. Thanks everyone.

[pcav]

First ticket opened: https://issues.qgis.org/issues/21372
New category Processing/OTB created
Please @rkanavath let me know your osgeo id, so I can add you as a default assignee.
Thanks.

[pcav]

Please check https://issues.qgis.org/projects/qgis/issues?category_id=167&set_filter=1&subproject_id=%21%2A

[spicysardine]

Thanks @rkanavath for the hint, now it works with: /usr/lib/x86_64-linux-gnu/otb/applications/ and /usr. Could you please add these standard paths to the man page, rather than your custom paths?
More generally: why is this necessary? Other backends do not need a config, and this is way easier for the user.

Worked for me too: Ubuntu 18.04. qgis 3.4.11 (ubuntugis stable)

[rkanavath]

It seems like debian package can update QGIS3.ini

$ grep OTB .local/share/QGIS/QGIS3/profiles/default/QGIS/QGIS3.ini
Configuration\OTB_ACTIVATE=true
Configuration\OTB_APP_FOLDER=/usr/lib/x86_64-linux-gnu/otb/applications
Configuration\OTB_FOLDER=/usr

Other providers search on some system paths to find an executable that indicates existence of a provider and windows users have OSGeo4W configure it for them. I think this should be managed as part of packaging and not in the code of every providers. QGIS3.ini could load / include other .ini files of provided and each provider manages their path using it's own .ini. saga.ini, otb.ini etc.. Packagers of Linux, Windows are free to fill up these .ini by their own. There could be 'qgis.d' that can have custom .ini files. This can also do many stuff to have a customized install of QGIS (just saying)

For instance,

• Debian/Ubuntu could use your configuration
• FreeBSD/OpenBSD could use some thing like /usr/local
• OSX could use something similar like Add OTB6 support to QGIS OSGeo/homebrew-osgeo4mac#431 depending on homebrew, macports etc..

I think otb team will be glad to help with otb.ini that can be used in debiangis and I am hoping other devs will do it too.. For instance, OTB standalone installer can distributre otb.ini and users will be much happy on both sides.

Providers that works without configuring anything is really nice both for QGIS and providers but adding list of linux/windows/osx/** OSes path is not a right way to do even though it is done.

Lastly, If discussion is to be continued, it must be moved to list or a new issue. A closed issue is not right place. I don't everybody will pay attention to a closed issue.