Add support for Netrc for Downloader

[sstadelman]

Implemented handling for netrc files. Matches connection settings for binary artifact host, and returns Basic authentication header to be appended to internal URLRequest of the Downloader. Will resolve Supporting basic auth for non-git binary dependency hosts.

Netrc parser based on Carthage implementation and tests, and migrated to regex for flexibility.

Features supported

• Parse Netrc string text: single and multi-line
• Match machine, login, password key values
• Ignore comments
• Ignore account and macdef entries (relevant only to ftp remotes)
• Accept any order of login, password, account key values
• Match default connection settings.
• Validates single instance of default connection setting, and that default connection setting is last entry
• Checks for netrc file in NSHomeDirectory()/.netrc by default
• Workspace supports custom netrcFilePath parameter

Sequence Flow (including SPM)

In SPM, a netrc file absolute path is resolved during (3) getActiveWorkspace() call of run() procedure, from (1) optional command argument. Optional absolute path is passed to initialized Workspace(5).

If binary artifacts should be updated, then the (7) download(...) operation attempts to load the netrc machine definitions from a file at the resolved netrcFilePath(8), or at NSHomeDirectory()/.netrc if nil. The optional resulting Netrc instance is passed to the created FoundationDownloader for each artifact (11). The downloader attempts to match a credential in the list of the Netrc instance's Machine definitions for the artifact host (13). If matched, the base64-encoded <user>:<password> pair is set to the Authorization header for the respective URLRequest.

Notes

1. The main question wrt API addition was: how to efficiently pass the netrc content to the Downloader protocol, which is lightweight by design. Though netrc is a common technique, it is not necessarily a universal solution--perhaps in the future we should also support application tokens, etc., which might similarly pollute the protocol. So, this implementation introduces the AuthorizationProviding interface, under which additional authentication techniques could in the future be added to the Workspace, and injected to the binary update process in Downloader implementations.

public protocol AuthorizationProviding {
    /// Optional `Authorization` header, likely added to `URLRequest`
    func authorization(for url: Foundation.URL) -> String?
}

Downloader is extended as follows:

/// The `Downloader` protocol abstract away the download of a file with a progress report.
public protocol Downloader {
    /**/
    /// Downloads a file and keeps the caller updated on the progress and completion.
    ///
    /// - Parameters:
    ///   - url: The `URL` to the file to download.
    ///   - destination: The `AbsolutePath` to download the file to.
    ///   - authorizationProvider: Optional provider supplying `Authorization` header to be added to `URLRequest`.
    ///   - progress: A closure to receive the download's progress as number of bytes.
    ///   - completion: A closure to be notifed of the completion of the download.
    func downloadFile(
        at url: Foundation.URL,
        to destination: AbsolutePath,
        withAuthorizationProvider authorizationProvider: AuthorizationProviding?,
        progress: @escaping Progress,
        completion: @escaping Completion
    )
}

2. The netrc credential injection is added to FoundationDownloader. Other Downloaders should implement handling according to their requirements.
3. Requires OSX 10.13. Will obligate swift-driver and swift-package-manager packages to raise minimum to 10.13 also. Discussed in thread.
4. An associated PR will be necessary in swift-package-manager to support --netrc-file option. However, the default NSHomeDirectory()/.netrc location for the netrc file can be supported immediately upon merging this PR in TSC.

[sstadelman]

@swift-ci test

[sstadelman]

@swift-ci test

[neonichu]

@swift-ci test

[sstadelman]

@swift-ci smoke-test

[sstadelman]

Mirrored in spm: swiftlang/swift-package-manager#2833.

[neonichu]

@swift-ci please test

[MaxDesiatov]

@sstadelman would you be able to resolve the conflict?

[sstadelman]

@MaxDesiatov conflict resolved.

[MaxDesiatov]

@swift-ci please test

[MaxDesiatov]

@swift-ci please smoke test

[MaxDesiatov]

@swift-ci please test

[MaxDesiatov]

The Linux build is failing:

error: reference to property 'downloads' in closure requires explicit 'self.' to make capture semantics explicit

[MaxDesiatov]

@swift-ci please test

[MaxDesiatov]

@swift-ci please test

[MaxDesiatov]

@sstadelman the build still fails on Linux, do you get the same error messages when you build and test on Linux with Swift 5.2 on your side? (The Linux Swift CI node uses Swift 5.2)

[sstadelman]

@MaxDesiatov it's a different failure message, regarding AbsolutePath optionality. Will test on Linux with 5.2 and ping you when complete. Sorry about that.

[sstadelman]

@MaxDesiatov resolved. passed on Swift:latest tag.

[MaxDesiatov]

@swift-ci please test

[MaxDesiatov]

@swift-ci please test

[sstadelman]

@swift-ci please test

[MaxDesiatov]

Awesome, thanks 👍

[MaxDesiatov]

@swift-ci please test

[MaxDesiatov]

@swift-ci please test

[MaxDesiatov]

Hi @neonichu, would you mind having a look at this? Thank you!

[neonichu]

This looks great, thanks @sstadelman for the implementation and @MaxDesiatov for helping with the review!

[sstadelman]

Thanks @neonichu and @MaxDesiatov for your assistance along the way!

@neonichu: what do you expect as the timeline for this being picked up by the standard Xcode distribution?

[MaxDesiatov]

Xcode 12 GM won't get this unless the SwiftPM release manager accepts this same PR to the release/5.3 branch. If not, it would be included in the version of Swift following after 5.3, whatever version that may be (and it's not announced yet as far as I know)

[OdNairy]

Xcode 12 GM Seed 1 (12A7209) doesn't support .netrc auth, right?

[MaxDesiatov]

I don't think this PR was re-submitted for the release/5.3 branch for that to happen, was it?

[sstadelman]

I don't think this PR was re-submitted for the release/5.3 branch for that to happen, was it?

No, is that the process for adoption in an upcoming patch?

[MaxDesiatov]

It doesn't guarantee that it will end up in a 5.3 patch release, but I guess it will be at least considered then if the change is not too disruptive and is well-tested. A similar situation happened to swiftlang/swift-package-manager#2905, which missed the 5.3.0 cut-off.

[sstadelman]

@MaxDesiatov what is the current status of this repository vis-a-vis swift-package-manager? Do all changes still need to be mirrored in both?

[MaxDesiatov]

Yes, definitely in the release/5.3 branch, and still in main until swiftlang/swift-package-manager#2898 is merged.