Deb external dependencies chicken and egg issue

[theangryangel]

Is your feature request related to a problem?

Python libraries, such as PyCurl depend on apt/deb dependencies, unfortunately, oca_install_addons gets into a chicken and egg issue where oca_list_external_dependencies won't run until the Python deps are installed, but the Python deps cannot be installed until libcurl4-(openssl|nss|gnutls)-dev apt package is installed.

i.e.

DEBIAN_FRONTEND=noninteractive apt-get install -qq --no-install-recommends $(oca_list_external_dependencies deb)
not found in addons path: component,component_event,connector,queue_job
Aborted!

Describe the solution you'd like

Either:

• There separate step beforehand that installs the deb dependencies first
• Or manifestoo requires a new sub-command to not throw errors?

Describe alternatives you've considered

N/A

Additional context

This seems to be a common issue where the workaround has been to modify the Dockerfile and add external dependencies, such as xmlsec1?

---

I am happy to do the leg work on this issue, but I'm not clear if it's an OCA-CI or a manifestoo issue, or both? Before I do any work I'd like some guidance or input in the better way to tackle this :)

[sbidoul]

Do you have a PR with the issue so I can see the context and the full logs?

[theangryangel]

I've created a minimal example at https://github.com/GlodoUK/oca-ci-issue-33. The failure at https://github.com/GlodoUK/oca-ci-issue-33/runs/7308997757?check_suite_focus=true specifically is my area of interest :)

Like I say, I'm happy to do the work, I'm just not sure if it's a manifestoo, or oca-ci issue, or a bit of both :)

[sbidoul]

I see. The deb external_dependencies in manifests are meant for runtime dependencies. Here some libraries are required at build time to compile pycurl which does not provide binary wheels on PyPI.

As you noticed, in this case we added the common dependencies in the ci image. So far for OCA it has not been too much of an issue in practice.

I'm a bit reluctant to add this library to the oca-ci image because then where do we draw the line?

Possible solutions that come to mind:

• create your own oca-ci derived image, but then I guess your next move will be to make it configurable in the oca-addons-repo-template :)
• add a question in the template to configure pre-install scripts, but the problem with that is that you'll need to add it in all repos that depend on the repo which needs the build time dependency

Not sure what's best...

Out of curiosity, why pycurl instead of a pure python library ?

[theangryangel]

The deb external_dependencies in manifests are meant for runtime dependencies.

Ah, that's my misunderstanding. It's obvious now you've pointed it out :)

I'm a bit reluctant to add this library to the oca-ci image because then where do we draw the line?

Fully understand, and this wasn't something I was expecting at all :)

add a question in the template to configure pre-install scripts, but the problem with that is that you'll need to add it in all repos that depend on the repo which needs the build time dependency

I'll see if we can prototype something and propose it over the next few days :) For now we've just added an extra step to our GitHub workflow.

Out of curiosity, why pycurl instead of a pure python library ?

We've had a scenario where we needed to talk to several different FTP servers directly from Odoo via a connector, and unfortunately every pure Python FTP library we tried had some form of problem.

The main pain point was around TLS and session resumption. Anything that ultimately was based on ftplib we specifically hit https://bugs.python.org/issue10808 and https://bugs.python.org/issue19500. Workarounds to manage the issue were viable for 1 FTP server and then would break another.

We needed a solution somewhat quickly and I reached for cURL because to date I've rarely had an issue.