When using s3 as a repo, rpm packages having special characters are not getting URL encoded

[frezbo]

I am using s3 as our repository. So if any packages have special characters, it fails to download as zypper does not URL encode the special characters and s3 denies the download.

[dmacvicar]

Can you point us to a repo example?

[frezbo]

@dmacvicar I don't have any public s3 repo to point out, but I can provide an example:
lets say I have an rpm named: containerd-0.2.8+gitr671_3addd8406531-58.1.x86_64.rpm, and the s3 generated link is https://s3.amazonaws.com/<bucket>/kubernetes/1.7/repos/containerd-0.2.8%2Bgitr671_3addd8406531-58.1.x86_64.rpm, so cleary the s3 URL is encoded, but zypper tries to download as https://s3.amazonaws.com/<bucket>/kubernetes/1.7/repos/containerd-0.2.8+gitr671_3addd8406531-58.1.x86_64.rpm, which throws an error as the URL is not encoded.

[mlandres]

[https://www.ietf.org/rfc/rfc1738.txt]
RFC 1738 Uniform Resource Locators (URL)

Thus, only alphanumerics, the special characters "$-_.+!*'(),", and
reserved characters used for their reserved purposes may be used
unencoded within a URL.

A '+' in the URL path is a safe char and does not need to be encoded. A download with '+' or '%2B' must deliver the same result.

@frezbo Without at least the /var/log/zypper.log of the failed command it's hard to tell what the actual problem is. You may open a bugreport at https://bugzilla.suse.com; use product openSUSE Tumbleweed and component libzypp; and attach the logfile there. I'll have a look at it.

[frezbo]

@mlandres thanks for the clarification, but this issue is specifically with s3. S3 required all special characters to be encoded.

[mlandres]

I found a few hints about this here, here and also in the aws documentations.

Looks like S3 is using an incorrect variant of URL-escaping, which we do not support. The + actually denotes a [SPACE] on S3, that's why they cant use a + literally. The aws documentation recommends to replace these "invalid" characters +=/ with these valid characters - (hyphen) _ (underscore) ~ (tilde), rather than %-encoding them.

So you could rename the affected files before creating the repo metadata and uploading the files to S3.

I will see whether we can support the broken S3 escaping in libzypp, but this may take a while. We can not simply advice our URL class to encode the +. This may indeed solve the download issue, but will break variable substitution where + has a special meaning...

[mlandres]

@sclausson: As @dmacvicar pointed out, libzypp itself needs to be modified first. The urlresolver plugin is currently used only to provide the baseurl pointing to the repositories root. Files below are then accessed by adapting the baseurls path. S3 however requires to sign each URL after the path has been modified, but currently there's no plugin API for doing this.

Once this is fixed, one could provide a python plugin script signing the URLs. Such a plugin could also workaround S3s broken URL encoding requirements (#143).

[mlandres]

Closing this issue in favor of #96. An enhanced urlresolver plugin would be able to handle the additional escapes required for S3.