Give the user an option to use predefined urls to download sdists from

[shubhbapna]

Fixes #138

User can provide a mapping of package to download source url in settings.yaml:

download_source:
  torch:
      url: "https://github.com/pytorch/pytorch/releases/download/v${version}/pytorch-v${version}.tar.gz"
      rename_to: "torch-${version}"

User can define a predefined url for a package from which all its sources will be downloaded from. Optionally they can rename the downloaded sdist to whatever they want.

The only supported template variables are: version. version is replaced by the version returned by the resolver.

[dhellmann]

Could you update https://github.com/python-wheel-build/fromager/blob/main/src/fromager/overrides.py#L131 as well, please, to include these new types of overrides.

[dhellmann]

It's not clear why this change is related to having the URLs. Maybe because we're assuming there are no sdists for those cases? That happens to be true for some of the examples we have.

[shubhbapna]

If the user is providing a url from which they want to download the sdist then the url that the resolver returns for the source dist won't be used anyways

[dhellmann]

Including or excluding specific formats influences the resolver, too. It will only return a version the distribution type we've asked for exists for that version.

I guess in most cases where we download source from an alternative location there isn't an sdist to use, so this is probably OK. We may end up wanting to expose options to control which types of distributions to consider, though. Let's deal with that in a separate PR. In fact, we may want to split the step for resolving the version from the step for downloading the source for that version.

It would be really good to see a functioning version of this PR with a real use case based on our downstream builds.

[shubhbapna]

Could you update https://github.com/python-wheel-build/fromager/blob/main/src/fromager/overrides.py#L131 as well, please, to include these new types of overrides.

Sorry, I didn't quite understand what I should add here?

[jwieleRH]

I was thinking of doing something like this myself for two reasons:

1. triton downloads the source from github in a file named by the version. This would be a reusable way to get the source and give the source archive a sensible name.
2. sacrebleu upstream builds its source distribution with setuptools, which leaves CHANGELOG.md out of the sdist which means you can't build the wheel from the sdist. Currently, we have a workaround that just creates an empty CHANGELOG.md file. Downloading the source from github instead of PyPI would allow us to build the wheel exactly the same way the upstream does, and of course the rename_to option would have the same application as with triton.

[dhellmann]

Including or excluding specific formats influences the resolver, too. It will only return a version the distribution type we've asked for exists for that version.

I guess in most cases where we download source from an alternative location there isn't an sdist to use, so this is probably OK. We may end up wanting to expose options to control which types of distributions to consider, though. Let's deal with that in a separate PR. In fact, we may want to split the step for resolving the version from the step for downloading the source for that version.

It would be really good to see a functioning version of this PR with a real use case based on our downstream builds.