Add OTB provider to qgis processing plugin [processing]

[rkanavath]

Description

This adds otb provider to qgis processing plugin. Here are some highlights.

• OTB descriptor files are not included/maintained in qgis source tree.
• OTB provider is generic and works for all OTB version starting from 6.6.0
• Testing of OTB algorithms has been added (using .yaml and others).
• No changes into qgis core other than a python import in Processing.py.
• Travis ci script (linux) is added to install otb package. (osx will be added if linux ci pipeline passes)

Checklist

• Commit messages are descriptive and explain the rationale for changes
• Commits which fix bugs include fixes #11111 in the commit message next to the description
• Commits which add new features are tagged with [FEATURE] in the commit message
• Commits which change the UI or existing user workflows are tagged with [needs-docs] in the commit message and contain sufficient information in the commit message to be documented
• I have read the QGIS Coding Standards and this PR complies with them
• This PR passes all existing unit tests (test results will be reported by travis-ci after opening this PR)
• New unit tests have been added for core changes
• I have run the scripts/prepare-commit.sh script before each commit

[rkanavath]

@luipir , could you have a look at changes. travis tests are fixed now

[luipir]

I'll give a try... but I'm busy on many front (userconf) in this moment. No problem if someone else is able to review before me

[nyalldawson]

I'd like to review before merge too, thanks!

But upfront: Thanks @rkanavath --- looks like great work from a preliminary read over 🎉

[luipir]

But upfront: Thanks @rkanavath --- looks like great work from a preliminary read over

of course, thanks @rkanavath :)

[rkanavath]

@nyalldawson , given that last commits are okay is it possible to move this to 3.6 milestone?

[pcav]

so the plugin is not trying to install, the use can do it in his usual way. thanks.

[rkanavath]

@pcav , yes plugin know nothing of how or if otb is installed. The change in travis docker is for testing in travis-ci.

[nyalldawson]

This looks good to me now. There's going to be some issues with interoperability with other algorithms in models due to the layer handling (i.e. not supporting memory layers), but that can always be addressed later and further refined.

@alexbruy did you want to review too?

There's still the outstanding question about the docker image changes too, which either @3nids or @elpaso will need to review

[SrNetoChan]

Wasn't it decided that except for GRASS and SAGA all the other providers would stay outside the Core, installed as plugins?

[rkanavath]

Thanks @nyalldawson and others for your help during review process. I really appreciate your inputs. Plus lot of great changes in processing which made code for provider much simpler!

I agree with you that in memory provider support can be added later. I will add this as a todo in otb bug tracker.

Regarding docker image changes, I don't have an alternative for testing otb. So will wait for inputs from @elpaso or @3nids.

[alexbruy]

@nyalldawson, sorry I won't review it. My position did not changed, this should be a 3rd party plugin like all other Processing providers. I don't see any reasons for exceptions, we had agreements "no more python plugins in the core" and "move processing providers outside core".

[pcav]

@alexbruy I can't find it back right now, but we have been discussing this case rather extensively, and the final decision was to include OTB

[alexbruy]

@pcav I perfectly remember what and how was decided

[nyalldawson]

@alexbruy

That's fine -- I totally get where you are coming from. If there's no objection to the docker changes I'll merge after thaw.

[rkanavath]

@nyalldawson, @pcav is it possible for someone else to review changes to docker file in travis to complete it. If @elpaso (I tried to contact him by mail) and @3nids too are not agreeing (don't want to understand why is that) to include otb provider and thereby refusing to even review changes what will be remaning option left.?

Even though docker changes aren't essential for functioning of provider itself but it allows for easy testing and helps improve integration and communication between two projects.

Thanks in advance.

[pcav]

I agree. Unfortunately, I don't know the docker infrastructure, so I cannot be of direct help here.

[nyalldawson]

It's ok, it's tagged as "merge after thaw", so we are just waiting on feature freeze to lift and I'll merge

[pcav]

Great news, I'll test soon after merge.
Thanks.

[vpoughon]

Hi all,

It's great to see that this PR is going to be merged. Thanks for all the work!

I'm a developer and PSC member of the OTB project, just a quick note: Reading this thread I see that there is a bit of friction about where this code belongs. IIRC, the main reason we pushed for the OTB provider to be part of QGIS core is that we had a few issues with API breakage. When it was outside of core the plugin suddenly stopped working for users who upgraded their QGIS version. If you can find another reference to a discution about this please let me know.

On our side, integration with QGIS is an important feature. We see a lot of users relying on both of these osgeo projects and we just want what's best for them. For this we are willing to spend ressources and work on QGIS integration in the years to come (i.e. assign dev time to make QGIS PRs, or whatever else is the best technical solution).

There's also the issue of documentation, and installation of the plugin. Both of these are a complete mess from a UX perspective given the state of QGIS/OTB/plugin version compatibility matrix.

Ultimately I feel like the question is about who should maintain this code. I'm not well versed in the technical details of this (@rkanavath did all the work), but I just want to make clear that we are not dumping this code in the QGIS repo hoping that it will just keep working. We are open for any discussion necessary (both technical and political) about how to make the best OTB-QGIS plugin for users. We have a gitlab issue tracker and a new discourse forum, both with github SSO, or feel free to reply here.

Thanks a lot

[pcav]

Hi Victor,
thanks for stepping in. I assume that if you take responsibility for this area of code, there is no reason not to include it. I fully support your efforts.

[pcav]

Compiled and tested: it requires to add manually the paths to OTB folder and OTB application folder. Tried /usr nd /usr/bin, but I still get errors.

[pcav]

No OTB algorithms found in '/usr/bin'. OTB will be disabled

[pcav]

ls /usr/bin/otbcli*
/usr/bin/otbcli /usr/bin/otbcli_MorphologicalClassification
/usr/bin/otbcli_BandMath /usr/bin/otbcli_MorphologicalMultiScaleDecomposition
/usr/bin/otbcli_BinaryMorphologicalOperation /usr/bin/otbcli_MorphologicalProfilesAnalysis
/usr/bin/otbcli_BlockMatching /usr/bin/otbcli_MultiImageSamplingRate
...