eve support datastore https certs

[naiming-zededa]

Signed-off-by: Naiming Shen naiming@zededa.com

• support for datastore cert chain from user configuration
• if the ds is local App based, skip verification for server cert

[eriknordmark]

Writing the cert to the ShareCertDirName means that other comunication such as other downloads will also trust the added cert, and here we just want to allow it for a particular download using that datastore.

Thus I think it is a lot safer to use the pattern which the zedcloud package uses, where we have a GetTlsConfig which loads certificates to get a tlsconfig struct, then this is used to form a transport and the transport is used to client := &http.Client{Transport: transport}
Here we have a ep.hClient which can be modified to have a particular TlsConfig with the specified certificate.
Alternatively, you can add an argument to httpClientSrcIP() to takes an optional certificate to use to create the TlsConfig. That ensures that the added cert only applies to this particular http client.

Also, presumably we should do this for sftp and the other transports; we have the same approach for all that we create a client with a source IP hence we should be able to specify the TlsConfig for all of them as well.

[naiming-zededa]

Writing the cert to the ShareCertDirName means that other comunication such as other downloads will also trust the added cert, and here we just want to allow it for a particular download using that datastore.

Thus I think it is a lot safer to use the pattern which the zedcloud package uses, where we have a GetTlsConfig which loads certificates to get a tlsconfig struct, then this is used to form a transport and the transport is used to client := &http.Client{Transport: transport}
Here we have a ep.hClient which can be modified to have a particular TlsConfig with the specified certificate.
Alternatively, you can add an argument to httpClientSrcIP() to takes an optional certificate to use to create the TlsConfig. That ensures that the added cert only applies to this particular http client.

Also, presumably we should do this for sftp and the other transports; we have the same approach for all that we create a client with a source IP hence we should be able to specify the TlsConfig for all of them as well.

Sure. I'll make the cert specific to the datastore attached for downloading. the sftp is not http/https related. we can have a separate story on that

[eriknordmark]

This is fine with me, but then we need to document that the certs in the UI are not added to the base Linux set of root CAs. Thus different than what we do for the proxy where we add the cert(s) to that base set.

[naiming-zededa]

right. in proxy case, there is a transport.Proxy item and also the proxy cert chain append into the linux certs directory. In this datastore case, the user explicitly want to have this cert chain to verify for this endpoint of https.

[eriknordmark]

Let it test

[eriknordmark]

Run eden tests again

[eriknordmark]

Run eden again

[eriknordmark]

Run run again

[giggsoff]

@naiming-zededa as I can see, we should also modify url for oci transport as well as sftp, because we split it

eve/pkg/pillar/cmd/downloader/syncop.go

Line 90 in cf9ed6e

serverURL, remoteName, err = ociRepositorySplit(dsCtx.DownloadURL)

Seems, something like should work

diff --git a/pkg/pillar/zedcloud/lookupproxy.go b/pkg/pillar/zedcloud/lookupproxy.go
index 03a75f6b6..59500f5e6 100644
--- a/pkg/pillar/zedcloud/lookupproxy.go
+++ b/pkg/pillar/zedcloud/lookupproxy.go
@@ -131,9 +131,10 @@ func IntfLookupProxyCfg(log *base.LogObject, status *types.DeviceNetworkStatus,
        // if download URL has "http://" or "https://" then no change here regardless of proxy
        // if there is proxy on this intf, treat empty url scheme as for https or http but prefer https,
        // replace the passed-in download-url scheme "docker://" and maybe "sftp://" later, to https
-       // XXX for sftp, currently the FQDN can not have scheme attached, but url.Parse will fail without it
-       if trType == zedUpload.SyncSftpTr && !strings.Contains(downloadURL, "://") {
-               downloadURL = "sftp://" + downloadURL
+       // XXX for sftp and oci, currently the FQDN can not have scheme attached, but url.Parse will fail without it
+       if (trType == zedUpload.SyncSftpTr || trType == zedUpload.SyncOCIRegistryTr) &&
+               !strings.Contains(downloadURL, "://") {
+               downloadURL = fmt.Sprintf("%s://%s", trType, downloadURL)
        }
        passURL, err := url.Parse(downloadURL)
        if err != nil {

[naiming-zededa]

@naiming-zededa as I can see, we should also modify url for oci transport as well as sftp, because we split it

eve/pkg/pillar/cmd/downloader/syncop.go

Line 90 in cf9ed6e

serverURL, remoteName, err = ociRepositorySplit(dsCtx.DownloadURL)

Seems, something like should work

diff --git a/pkg/pillar/zedcloud/lookupproxy.go b/pkg/pillar/zedcloud/lookupproxy.go
index 03a75f6b6..59500f5e6 100644
--- a/pkg/pillar/zedcloud/lookupproxy.go
+++ b/pkg/pillar/zedcloud/lookupproxy.go
@@ -131,9 +131,10 @@ func IntfLookupProxyCfg(log *base.LogObject, status *types.DeviceNetworkStatus,
        // if download URL has "http://" or "https://" then no change here regardless of proxy
        // if there is proxy on this intf, treat empty url scheme as for https or http but prefer https,
        // replace the passed-in download-url scheme "docker://" and maybe "sftp://" later, to https
-       // XXX for sftp, currently the FQDN can not have scheme attached, but url.Parse will fail without it
-       if trType == zedUpload.SyncSftpTr && !strings.Contains(downloadURL, "://") {
-               downloadURL = "sftp://" + downloadURL
+       // XXX for sftp and oci, currently the FQDN can not have scheme attached, but url.Parse will fail without it
+       if (trType == zedUpload.SyncSftpTr || trType == zedUpload.SyncOCIRegistryTr) &&
+               !strings.Contains(downloadURL, "://") {
+               downloadURL = fmt.Sprintf("%s://%s", trType, downloadURL)
        }
        passURL, err := url.Parse(downloadURL)
        if err != nil {

@giggsoff Right, i did change those in my workspace, but i'm waiting for eden error we have seen on 'route unreachable', any ida?

[eriknordmark]

LGTM. Let's run eden tests once more