metalink: remove

[bagder]

Warning: this will make existing curl command lines that use metalink to
stop working.

Reasons for removal:

1. We've found several security problems and issues involving the
   metalink support in curl. The issues are not detailed here. When
   working on those, it become apparent to the team that several of the
   problems are due to the system design, metalink library API and what
   the metalink RFC says. They are very hard to fix on the curl side
   only.

2. The metalink usage with curl was only very briefly documented and was
   not following the "normal" curl usage pattern in several ways, making
   it surprising and non-intuitive which could lead to further security
   issues.

3. The metalink library was last updated 6 years ago and wasn't so
   active the years before that either. An unmaintained library means
   there's a security problem waiting to happen. This is probably reason
   enough.

4. Metalink requires an XML parsing library, which is complex code (even
   the smaller alternatives) and to this day often gets security
   updates.

5. Metalink is not a widely used curl feature. In the 2020 curl user
   survey, only 1.4% of the responders said that they'd are using
   it. Searching the web also show very few traces of it being used,
   even with other tools.

6. The torrent format and associated technology clearly won for
   downloading large files from multiple sources in parallel.

[ytrezq]

I disagree on point 6 though, torrent will never support e2dk or gnunet or ipts unlike MetaLink. But I recognize curl only supports servers protocols and not p2p ones.

[bagder]

Assuming no major obstacle, I will merge this on June 7th.

[kdudka]

I would prefer if --with-libmetalink made the configure script fail rather than being silently ignored. If someone expects metalink to be available, they should learn about its removal rather sooner than later.

[bagder]

I would prefer if --with-libmetalink made the configure script fail rather than being silently ignored.

Good idea!

[danielgustafsson]

Another idea could be to move metalink support to being an experimental feature (even though experimental generally means moving the other way) requiring an explicit opt-in before removing it. Iff there is an outcry after the release then there is a way for powerusers to keep their systems running while we take the discussion on how to proceed. FWIW, I'm not sure I want this but I wanted to raise it in the discussion before the PR is merged.

[bagder]

I don't think we should merge even an experimental feature with known security problems so they would still need to be worked on first.

Power users can just revert this removal to get metalink back with a local patch. If there's a strong enough demand to get it back "for real", I expect developer time to get funded to fix the issues.

[danielgustafsson]

On 3 Jun 2021, at 14:44, Daniel Stenberg ***@***.***> wrote: I don't think we should merge even an experimental feature with known security problems so they would still need to be worked on first. Power users can just revert this removal to get metalink back with a local patch. If there's a strong enough demand to get it back "for real", I expect developer time to get funded to fix the issues.

Agreed, I mainly wanted the discussion to have covered all angles given how drastic this measure is.

[xquery]

maybe this has already been discussed here or elsewhere - apologies

I have no sense of metalink usage ( I see very few respondents to survey use it) ... adhoc observation is that there is not a lot of adoption ... in the best of worlds (good developer support of exotic feature) my pref would be that said feature would not be part of the core ... considering we have less then good implementation - its a 'no brainer' to deprecate.

Alternately is it time to discuss (prob again) how to formalise pluggability beyond obvious routes .. it makes my brain hurt as its so easy to extend curl/libcurl and not warrant inclusion into core repo ... but is it getting to the point where we need more points of extensibility ?

[bagder]

is it getting to the point where we need more points of extensibility ?

Perhaps. I don't think this is the last straw that breaks the camel's back though because I don't think it is popular enough.

Extensibility, plugins and add-ons are mentioned every once in a while. It's just that nobody has yet even tried to write up a suggestion of how it could work - at any level of ambition - so I think we're still not quite there yet. Personally, I'm casually skeptic because I think both curl and libcurl are what they are partly because they provide feature-packed all-in-one packages.

[bagder]

Closed by 265b14d

[sjehuda]

Please. Pardon for responding to this now.

Warning: this will make existing curl command lines that use metalink to stop working.

Reasons for removal:

1. We've found several security problems and issues involving the
   metalink support in curl. The issues are not detailed here. When
   working on those, it become apparent to the team that several of the
   problems are due to the system design, metalink library API and what
   the metalink RFC says. They are very hard to fix on the curl side
   only.

Is this still the current state of affairs?

2. The metalink usage with curl was only very briefly documented and was
   not following the "normal" curl usage pattern in several ways, making
   it surprising and non-intuitive which could lead to further security
   issues.

I am willing to fullfil the task of documentation.

3. The metalink library was last updated 6 years ago and wasn't so
   active the years before that either. An unmaintained library means
   there's a security problem waiting to happen. This is probably reason
   enough.

Whch project is it?

Can we take over that project?

4. Metalink requires an XML parsing library, which is complex code (even
   the smaller alternatives) and to this day often gets security
   updates.

Please elaborate, so I would know better about this specific matter.

5. Metalink is not a widely used curl feature. In the 2020 curl user
   survey, only 1.4% of the responders said that they'd are using
   it. Searching the web also show very few traces of it being used,
   even with other tools.

In my collage the usage is almost exclusive.

I have also wrote about the reasons which make it better than torrent.

gemini://woodpeckersnest.space/~schapps/journal/2024-12-27-metalink-versus-torrent.gmi

6. The torrent format and associated technology clearly won for
   downloading large files from multiple sources in parallel.

No it has not. I think, that it might appear to be so, to those who utilize HTML browsers and are exposed to HTTP based link indexers (i.e. search engine, so called) and publishing platforms.

In the last week, I have recovered 3,843 GB of thought to be lost data has been recovered within less than a week via eD2k and Gnutella; data that is no longer or hardly accessible via BitTorrent because of the double dependency of existance of a specific file-set-order and torrent-metadata files.

Obviously, BitTorrent would lose over a period of 50, 100, 200 and 1000 of years, against systems such as eD2k and Gnutella.

In various of forums Metalink is utilized to share data that is available on ADC, DC, eD2k and Gnutella.

I plea to reconsider Metalink, because it is in the best interest of those that want to reduce access to information to rely on lesser stable sharing means such as Torrent files that refer to themselves over P2P, instead of Metalinks that actually refer to actual files.

Of note

The popularity of Metalinks relies also on software.

If no software would support Metalink, then no man would utilize it.

There may be 10 software to support Metalink; each software is built by 20 people.

There may be 100 of million of people who desire to utilize Metalink.

The power of 200 (10x20) people is crucial against 100 of million.

[bagder]

You're commenting on a 3.5 years old closed PR here. I would propose you start a new discussion thread instead.

Is this still the current state of affairs?

We dropped metalink back then yes and nobody has missed it since. It is all in the rear view mirror now.

Which project is it?

https://github.com/metalink-dev/libmetalink

Can we take over that project?

You need to ask the owners of it.

Metalink requires an XML parsing library, which is complex code (even the smaller alternatives) and to this day often gets security updates.
Please elaborate, so I would know better about this specific matter.

What about it is not clear?

I plea to reconsider Metalink

Given the number of problems and challenges involved I think it is unlikely that we would accept a PR bringing it back, but if everything was done properly, documented, tested etc then maybe. I am not at all convinced anyone is willing and able to do what's necessary to make such a PR.

[sjehuda]

I have strted a duscussion at #15818 and open a ticket at metalink-dev/libmetalink#5