curl package upgraded to 7.81.0

[ghost]

Changelog category (leave one):

• Not for changelog (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
To fix vulnerabilities reported by WhiteSource

Information about CI checks: https://clickhouse.tech/docs/en/development/continuous-integration/

[alexey-milovidov]

LGTM.

But the proper solution will be to remove curl.
It is used only to send data to sentry. We should replace it to something minimal or remove altogether.

[alexey-milovidov]

@Meena-Renganathan It does not build on Mac OS X,
see the "special build check".

[alexey-milovidov]

@Meena-Renganathan Let's fix the build or close this.

[ghost]

Sorry, I'm working on it. I will make the changes to fix the build error soon.

[ghost]

Hi @alexey-milovidov, I've tried to reproduce the error by building curl (with openssl option) in my local MacOS but it's getting built successfully. Here is the details of my MacOS:

macOS Monterey
Clang Version 13.

CC libcurl_la-hostip.lo
CC libcurl_la-hostip4.lo
CC libcurl_la-hostip6.lo
CC libcurl_la-hostsyn.lo

Could you help how it can take it up further to get this fixed?

[alexey-milovidov]

Let's check the build log: https://s3.amazonaws.com/clickhouse-builds/35130/d19090fdaf89162ba5527f7f376394a6cdd4081f/binary_darwin/build_log.log

Maybe it is related to some defines.
We can investigate it by running exactly the same compiler command.

Also another suggestion: maybe remove curl instead of updating it?

[ghost]

Thank you @alexey-milovidov. But, while trying upgrade the curl package to 7.81.0, observed the curl is getting referenced by sentry-native and azure modules. So, decided to upgrade the curl.

[ghost]

Thank you @alexey-milovidov. But, while trying upgrade the curl package to 7.81.0, observed the curl is getting referenced by sentry-native and azure modules. So, decided to upgrade the curl.

[alexey-milovidov]

Is it possible to remove curl from Azure?

[ghost]

It is not possible to directly remove the curl dependency from azure.

Azure module uses the libcurl and WinHTTP libraries as HTTP stacks for communicating with Azure services over the network. When no specific configuration is set, the default transport adapter will be selected based on the OS. But custom transport adapter needs to be implemented to avoid the dependency on curl.

[alexey-milovidov]

Ok, then we need to fix this remaining build.

PS. There was similar problem in AWS library. They used curl and it led to buffering all data in memory (I believe that it is not a downside of a curl, it is actually a good, very well tested library), and we replaced it by Poco. Maybe do similar with Azure?

[alexey-milovidov]

Still not fixed.

[ghost]

Hi @alexey-milovidov, yeah noticed the error and working on it to fix.

[alexey-milovidov]

We don't allow find_library in our cmake files.

[ghost]

Hi @alexey-milovidov, any specific reason we are restricting the find_library in cmake files?

[alexey-milovidov]

It is needed for hermetic builds - to avoid dependence on the environment.

[alexey-milovidov]

If the new curl version started to need something from SystemConfiguration,
maybe better to chop it from the library?

[alexey-milovidov]

I did not check, but need to figure out where exactly these dependencies are needed? Is it possible to avoid them?

[alexey-milovidov]

@Meena-Renganathan
Undefining this macro would help:
https://github.com/curl/curl/blob/8a1fa3b364b9db52cf3e8d843ac1e749890985e6/lib/hostip.c#L71

[ghost]

Hi @alexey-milovidov,

To get better clarity, underlining the macro ENABLE_IPV6 doesn't allow to support IPV6 addressing format in ClickHouse. Is that my understanding, correct?

[alexey-milovidov]

No. We must support IPv6.

[alexey-milovidov]

CURL_OSX_CALL_COPYPROXIES

This.

I don't know what is "copy proxies" on OS X, and so we don't need it.

[alexey-milovidov]

Also please grep the code base for all instances of CURL_OSX_*.

[ghost]

Undefining this macro would require a code change in curl repo (curl/lib/curl_setup.h).

[alexey-milovidov]

Another option is to disable curl, azure and sentry on Mac OS, so the problem will not exist.

[alexey-milovidov]

Also getting rid of curl is a good option because we don't need it.
It can be done easily by implementing Azure::Core::Http::HttpTransport.

[ghost]

I'm planning to proceed with the option of disabling curl, azure and sentry on Mac OS.

[alexey-milovidov]

@Mergifyio update

[mergify]

update

❌ Pull request can't be updated with latest base branch changes

Mergify needs the author permission to update the base branch of the pull request.
DevTeamBK needs to authorize modification on its head branch.
err-code: 23BB7

[alexey-milovidov]

This is ready to be merged.
Need to merge with master one more time.

[alexey-milovidov]

Ok, let's try as is.

[azat]

Let's check the build log: https://s3.amazonaws.com/clickhouse-builds/35130/d19090fdaf89162ba5527f7f376394a6cdd4081f/binary_darwin/build_log.log

Looks like the problem with bool type.

[alexey-milovidov]

ClickHouse build check (actions) — 22/22 builds are OK
ClickHouse special build check (actions) — 8/8 builds are OK