Cannot copy or deploy images behind proxy #2147

Closed
ptylenda opened this Issue Jun 23, 2016 · 9 comments

Comments

Projects
None yet
3 participants

Required information

  • Distribution:
Distributor ID: Ubuntu
Description:    Ubuntu 16.04 LTS
Release:        16.04
Codename:       xenial
  • The output of "lxc info" or if that fails:
apicompat: 0
auth: trusted
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIIFyzCCA7OgAwIBAgIQYfGZKArrqOuyizIJ2FeASzANBgkqhkiG9w0BAQsFADA9
    MRwwGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMR0wGwYDVQQDDBRyb290QGV2
    ZXJ5ZGF5LWFuZHJpYTAeFw0xNjA2MjMxNTA1NDlaFw0yNjA2MjExNTA1NDlaMD0x
    HDAaBgNVBAoTE2xpbnV4Y29udGFpbmVycy5vcmcxHTAbBgNVBAMMFHJvb3RAZXZl
    cnlkYXktYW5kcmlhMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAsbL8
    L+NrBEdofCzP21qk0ZvXSXW/iuh8bYOoMKOoYpTquC+g+sXWXFof0g5iVUUCj1pO
    VLX7ubulzXKSdUxZhkJ+0/COcLJiR4KYglhrMythzwk1l80Jro92q3m/i5j0GnZe
    Ta3etTDKS22DR8upIKN5eS6zVHSQ6MfOEqK2GyPDRf7ZKYC1MrHQavCFVTPJLIG4
    vO4ND3ESy1gm45YvUxcLAM1JEjH+D4s7Ip+i1/o5frY1THlO9lDnmI5cTnv3/wG1
    oHD3rinht2baODqj1vbBih6l7q5uU/gfT+54IHxAKTf9c46c7AWnJqbKtZ6vKLuG
    1lcTxO/c9tzuwXiNO57jg+JV2i3PncrGRRGMLJPM0EEiqfS5m9bGpA1bpfcR6gDH
    DRqJ3YFe2n49aGavehecSSAgjTpeSa3X9TYkqnN51Dlmt/QkDlOdGs2ENq/vjpxT
    Kbo34X2YKngx5OLhp2sED6wQg8bK5lO3Lw3nv/uyX5KNiNsCpH72h0x9Of7PRQN/
    zFnFxKfP1fuyn/jpFZN1BiXyXUMMPG1HLz/1HNhGVOuJ/mj13WqOT1H/9Ulwth/L
    ZPiSc9z7LWjJ2iBjcRQ5Hex7TQYI9egHBjZk0WAQSGQgAAYpUhUNEf2yZSjQZSCn
    9fJcqHoR+clIFTqAfD/cZtpeWa5JovUetv6LO+ECAwEAAaOBxjCBwzAOBgNVHQ8B
    Af8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADCBjQYD
    VR0RBIGFMIGCgg9ldmVyeWRheS1hbmRyaWGCDjE5Mi4xNjguMC4yLzI0ghpmZTgw
    OjoyMzA6NDhmZjpmZTMzOmMzOC82NIIcZmU4MDo6YThjMDpmNWZmOmZlNjY6NWM5
    MS82NIIZZmU4MDo6Zjo4MmZmOmZlNTI6NjQzNC82NIIKZmU4MDo6MS82NDANBgkq
    hkiG9w0BAQsFAAOCAgEAl4jlGrgFjJNhrJlfiCXeDFMk6qGNbkTKCbChMSgpVk1X
    5UUSsTpBnHJW58u5p+ObAdpJOF8JotyBzgAeoLYRyLL4mhVHymUkX5jQXEkMsiDP
    unYpUQYbo8yeu2Fu/RQcOi3+VkUtPUEH71c77VJ+Ce2eshoYd18P3hCnANmyJbhC
    IswDU2AV/34FRGQgET0+105qb08aABShCF4dcBjXuM5SZKtJ88+7GAOA5ZTWH9OL
    N2VgNQi3jLS9L7/58DMl8oKrk5qNgJzuKpSRxJMmRNyZryD2nMZhGEbvC0d6Zqcs
    /BrXJYLYM1N5ieTWCZmLehUbGa99qqDA9DdPbSwejyose1yvPrsX1EjkAixl4rhd
    J2iMOtXPKjo1QchfHfTFLk60psa1trkcAClix/0WL3ybIe+3FMbKr2zA9+ZIOsfh
    i0hQQHxlyCRIASoYuEhtYpM+jV0MRjDfrRToK6FiF3hPmJGayrP1Zm+anwTaNkBj
    1trnK25gGK7iS1xHTg2TGRFNNDLpXrH2SaGOGNzRYTYzfrQhGX7SC8CIAUohDg4g
    Lg+4NUTyqwCgqaU9h8kFXtNO09aJLz9HYvO6GWhGECN46LjkvaU6PakQWaf0tJkB
    858sdyjMaF7aS/BXmjxUGxNjKYB11YfyiMB3RLhdni3oolxLIjdwyOucCtXyWf0=
    -----END CERTIFICATE-----
  driver: lxc
  driverversion: 2.0.1
  kernel: Linux
  kernelarchitecture: x86_64
  kernelversion: 4.4.0-24-generic
  server: lxd
  serverpid: 11555
  serverversion: 2.0.2
  storage: dir
  storageversion: ""
config: {}
public: false

Issue description

For some reason images cannot be copied when being behind proxy. Proxy settings are set properly according to https://github.com/lxc/lxd/blob/master/doc/configuration.md

Steps to reproduce

  1. My $HTTP_PROXY and $HTTPS_PROXY env vars are set properly
  2. My images sources:
ubuntu@everyday-andria:~$ sudo lxc remote list
+-----------------+------------------------------------------+---------------+--------+--------+
|      NAME       |                   URL                    |   PROTOCOL    | PUBLIC | STATIC |
+-----------------+------------------------------------------+---------------+--------+--------+
| images          | https://images.linuxcontainers.org       | lxd           | YES    | NO     |
+-----------------+------------------------------------------+---------------+--------+--------+
| local (default) | unix://                                  | lxd           | NO     | YES    |
+-----------------+------------------------------------------+---------------+--------+--------+
| ubuntu          | https://cloud-images.ubuntu.com/releases | simplestreams | YES    | YES    |
+-----------------+------------------------------------------+---------------+--------+--------+
| ubuntu-daily    | https://cloud-images.ubuntu.com/daily    | simplestreams | YES    | YES    |
+-----------------+------------------------------------------+---------------+--------+--------+
  1. Trying to launch a container with xenial (with debug):
ubuntu@everyday-andria:~$ sudo lxc launch ubuntu:14.04 t1 --debug
DBUG[06-23|16:05:17] Raw response: {"type":"sync","status":"Success","status_code":200,"metadata":{"api_extensions":[],"api_status":"stable","api_version":"1.0","auth":"trusted","config":{},"environment":{"addresses":[],"architectures":["x86_64","i686"],"certificate":"-----BEGIN CERTIFICATE-----\nMIIFyzCCA7OgAwIBAgIQYfGZKArrqOuyizIJ2FeASzANBgkqhkiG9w0BAQsFADA9\nMRwwGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMR0wGwYDVQQDDBRyb290QGV2\nZXJ5ZGF5LWFuZHJpYTAeFw0xNjA2MjMxNTA1NDlaFw0yNjA2MjExNTA1NDlaMD0x\nHDAaBgNVBAoTE2xpbnV4Y29udGFpbmVycy5vcmcxHTAbBgNVBAMMFHJvb3RAZXZl\ncnlkYXktYW5kcmlhMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAsbL8\nL+NrBEdofCzP21qk0ZvXSXW/iuh8bYOoMKOoYpTquC+g+sXWXFof0g5iVUUCj1pO\nVLX7ubulzXKSdUxZhkJ+0/COcLJiR4KYglhrMythzwk1l80Jro92q3m/i5j0GnZe\nTa3etTDKS22DR8upIKN5eS6zVHSQ6MfOEqK2GyPDRf7ZKYC1MrHQavCFVTPJLIG4\nvO4ND3ESy1gm45YvUxcLAM1JEjH+D4s7Ip+i1/o5frY1THlO9lDnmI5cTnv3/wG1\noHD3rinht2baODqj1vbBih6l7q5uU/gfT+54IHxAKTf9c46c7AWnJqbKtZ6vKLuG\n1lcTxO/c9tzuwXiNO57jg+JV2i3PncrGRRGMLJPM0EEiqfS5m9bGpA1bpfcR6gDH\nDRqJ3YFe2n49aGavehecSSAgjTpeSa3X9TYkqnN51Dlmt/QkDlOdGs2ENq/vjpxT\nKbo34X2YKngx5OLhp2sED6wQg8bK5lO3Lw3nv/uyX5KNiNsCpH72h0x9Of7PRQN/\nzFnFxKfP1fuyn/jpFZN1BiXyXUMMPG1HLz/1HNhGVOuJ/mj13WqOT1H/9Ulwth/L\nZPiSc9z7LWjJ2iBjcRQ5Hex7TQYI9egHBjZk0WAQSGQgAAYpUhUNEf2yZSjQZSCn\n9fJcqHoR+clIFTqAfD/cZtpeWa5JovUetv6LO+ECAwEAAaOBxjCBwzAOBgNVHQ8B\nAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADCBjQYD\nVR0RBIGFMIGCgg9ldmVyeWRheS1hbmRyaWGCDjE5Mi4xNjguMC4yLzI0ghpmZTgw\nOjoyMzA6NDhmZjpmZTMzOmMzOC82NIIcZmU4MDo6YThjMDpmNWZmOmZlNjY6NWM5\nMS82NIIZZmU4MDo6Zjo4MmZmOmZlNTI6NjQzNC82NIIKZmU4MDo6MS82NDANBgkq\nhkiG9w0BAQsFAAOCAgEAl4jlGrgFjJNhrJlfiCXeDFMk6qGNbkTKCbChMSgpVk1X\n5UUSsTpBnHJW58u5p+ObAdpJOF8JotyBzgAeoLYRyLL4mhVHymUkX5jQXEkMsiDP\nunYpUQYbo8yeu2Fu/RQcOi3+VkUtPUEH71c77VJ+Ce2eshoYd18P3hCnANmyJbhC\nIswDU2AV/34FRGQgET0+105qb08aABShCF4dcBjXuM5SZKtJ88+7GAOA5ZTWH9OL\nN2VgNQi3jLS9L7/58DMl8oKrk5qNgJzuKpSRxJMmRNyZryD2nMZhGEbvC0d6Zqcs\n/BrXJYLYM1N5ieTWCZmLehUbGa99qqDA9DdPbSwejyose1yvPrsX1EjkAixl4rhd\nJ2iMOtXPKjo1QchfHfTFLk60psa1trkcAClix/0WL3ybIe+3FMbKr2zA9+ZIOsfh\ni0hQQHxlyCRIASoYuEhtYpM+jV0MRjDfrRToK6FiF3hPmJGayrP1Zm+anwTaNkBj\n1trnK25gGK7iS1xHTg2TGRFNNDLpXrH2SaGOGNzRYTYzfrQhGX7SC8CIAUohDg4g\nLg+4NUTyqwCgqaU9h8kFXtNO09aJLz9HYvO6GWhGECN46LjkvaU6PakQWaf0tJkB\n858sdyjMaF7aS/BXmjxUGxNjKYB11YfyiMB3RLhdni3oolxLIjdwyOucCtXyWf0=\n-----END CERTIFICATE-----\n","driver":"lxc","driver_version":"2.0.1","kernel":"Linux","kernel_architecture":"x86_64","kernel_version":"4.4.0-24-generic","server":"lxd","server_pid":11555,"server_version":"2.0.2","storage":"dir","storage_version":""},"public":false}}

DBUG[06-23|16:05:17] Raw response: {"type":"sync","status":"Success","status_code":200,"metadata":{"api_extensions":[],"api_status":"stable","api_version":"1.0","auth":"trusted","config":{},"environment":{"addresses":[],"architectures":["x86_64","i686"],"certificate":"-----BEGIN CERTIFICATE-----\nMIIFyzCCA7OgAwIBAgIQYfGZKArrqOuyizIJ2FeASzANBgkqhkiG9w0BAQsFADA9\nMRwwGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMR0wGwYDVQQDDBRyb290QGV2\nZXJ5ZGF5LWFuZHJpYTAeFw0xNjA2MjMxNTA1NDlaFw0yNjA2MjExNTA1NDlaMD0x\nHDAaBgNVBAoTE2xpbnV4Y29udGFpbmVycy5vcmcxHTAbBgNVBAMMFHJvb3RAZXZl\ncnlkYXktYW5kcmlhMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAsbL8\nL+NrBEdofCzP21qk0ZvXSXW/iuh8bYOoMKOoYpTquC+g+sXWXFof0g5iVUUCj1pO\nVLX7ubulzXKSdUxZhkJ+0/COcLJiR4KYglhrMythzwk1l80Jro92q3m/i5j0GnZe\nTa3etTDKS22DR8upIKN5eS6zVHSQ6MfOEqK2GyPDRf7ZKYC1MrHQavCFVTPJLIG4\nvO4ND3ESy1gm45YvUxcLAM1JEjH+D4s7Ip+i1/o5frY1THlO9lDnmI5cTnv3/wG1\noHD3rinht2baODqj1vbBih6l7q5uU/gfT+54IHxAKTf9c46c7AWnJqbKtZ6vKLuG\n1lcTxO/c9tzuwXiNO57jg+JV2i3PncrGRRGMLJPM0EEiqfS5m9bGpA1bpfcR6gDH\nDRqJ3YFe2n49aGavehecSSAgjTpeSa3X9TYkqnN51Dlmt/QkDlOdGs2ENq/vjpxT\nKbo34X2YKngx5OLhp2sED6wQg8bK5lO3Lw3nv/uyX5KNiNsCpH72h0x9Of7PRQN/\nzFnFxKfP1fuyn/jpFZN1BiXyXUMMPG1HLz/1HNhGVOuJ/mj13WqOT1H/9Ulwth/L\nZPiSc9z7LWjJ2iBjcRQ5Hex7TQYI9egHBjZk0WAQSGQgAAYpUhUNEf2yZSjQZSCn\n9fJcqHoR+clIFTqAfD/cZtpeWa5JovUetv6LO+ECAwEAAaOBxjCBwzAOBgNVHQ8B\nAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADCBjQYD\nVR0RBIGFMIGCgg9ldmVyeWRheS1hbmRyaWGCDjE5Mi4xNjguMC4yLzI0ghpmZTgw\nOjoyMzA6NDhmZjpmZTMzOmMzOC82NIIcZmU4MDo6YThjMDpmNWZmOmZlNjY6NWM5\nMS82NIIZZmU4MDo6Zjo4MmZmOmZlNTI6NjQzNC82NIIKZmU4MDo6MS82NDANBgkq\nhkiG9w0BAQsFAAOCAgEAl4jlGrgFjJNhrJlfiCXeDFMk6qGNbkTKCbChMSgpVk1X\n5UUSsTpBnHJW58u5p+ObAdpJOF8JotyBzgAeoLYRyLL4mhVHymUkX5jQXEkMsiDP\nunYpUQYbo8yeu2Fu/RQcOi3+VkUtPUEH71c77VJ+Ce2eshoYd18P3hCnANmyJbhC\nIswDU2AV/34FRGQgET0+105qb08aABShCF4dcBjXuM5SZKtJ88+7GAOA5ZTWH9OL\nN2VgNQi3jLS9L7/58DMl8oKrk5qNgJzuKpSRxJMmRNyZryD2nMZhGEbvC0d6Zqcs\n/BrXJYLYM1N5ieTWCZmLehUbGa99qqDA9DdPbSwejyose1yvPrsX1EjkAixl4rhd\nJ2iMOtXPKjo1QchfHfTFLk60psa1trkcAClix/0WL3ybIe+3FMbKr2zA9+ZIOsfh\ni0hQQHxlyCRIASoYuEhtYpM+jV0MRjDfrRToK6FiF3hPmJGayrP1Zm+anwTaNkBj\n1trnK25gGK7iS1xHTg2TGRFNNDLpXrH2SaGOGNzRYTYzfrQhGX7SC8CIAUohDg4g\nLg+4NUTyqwCgqaU9h8kFXtNO09aJLz9HYvO6GWhGECN46LjkvaU6PakQWaf0tJkB\n858sdyjMaF7aS/BXmjxUGxNjKYB11YfyiMB3RLhdni3oolxLIjdwyOucCtXyWf0=\n-----END CERTIFICATE-----\n","driver":"lxc","driver_version":"2.0.1","kernel":"Linux","kernel_architecture":"x86_64","kernel_version":"4.4.0-24-generic","server":"lxd","server_pid":11555,"server_version":"2.0.2","storage":"dir","storage_version":""},"public":false}}

DBUG[06-23|16:05:17] Posting {"name":"t1","source":{"certificate":"","fingerprint":"14.04","mode":"pull","protocol":"simplestreams","server":"https://cloud-images.ubuntu.com/releases","type":"image"}}
 to http://unix.socket/1.0/containers
DBUG[06-23|16:05:17] Raw response: {"type":"async","status":"Operation created","status_code":100,"metadata":{"id":"f475bbcd-1f37-4360-8545-0e246f964593","class":"task","created_at":"2016-06-23T16:05:17.716242496Z","updated_at":"2016-06-23T16:05:17.716242496Z","status":"Running","status_code":103,"resources":{"containers":["/1.0/containers/t1"]},"metadata":null,"may_cancel":false,"err":""},"operation":"/1.0/operations/f475bbcd-1f37-4360-8545-0e246f964593"}

Creating t1
DBUG[06-23|16:05:17] 1.0/operations/f475bbcd-1f37-4360-8545-0e246f964593/wait
DBUG[06-23|16:05:37] Raw response: {"type":"sync","status":"Success","status_code":200,"metadata":{"id":"f475bbcd-1f37-4360-8545-0e246f964593","class":"task","created_at":"2016-06-23T16:05:17.716242496Z","updated_at":"2016-06-23T16:05:17.716242496Z","status":"Failure","status_code":400,"resources":{"containers":["/1.0/containers/t1"]},"metadata":null,"may_cancel":false,"err":"Get https://cloud-images.ubuntu.com/releases/streams/v1/index.json: Unable to connect to: cloud-images.ubuntu.com:443"}}

error: Get https://cloud-images.ubuntu.com/releases/streams/v1/index.json: Unable to connect to: cloud-images.ubuntu.com:443
  1. When doing curl (without any additional proxy settings, just using these from environment):
curl  https://cloud-images.ubuntu.com/releases/streams/v1/index.json
{
 "index": {
  "com.ubuntu.cloud:released:joyent": {
   "updated": "Wed, 13 Apr 2016 14:14:06 +0000",
   "clouds": [
    {
     "region": "eu-ams-1",
     "endpoint": "https://eu-ams-1.api.joyentcloud.com"
    },
    {
     "region": "us-sw-1",
     "endpoint": "https://us-sw-1.api.joyentcloud.com"
    },
.......

Configuration https://github.com/lxc/lxd/blob/master/doc/configuration.md says, that there is no need for setting lxd core.proxy_https, but even if I set it, it still doesn't work...

I have tried solution described in #1909, but even if I have profile like below, it still doesn't work:

ubuntu@everyday-andria:~$ sudo lxc profile show default
name: default
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
Owner

stgraber commented Jun 23, 2016

cloud-images.ubuntu.com is currently offline due to network issues on the Canonical network.

@stgraber stgraber closed this Jun 23, 2016

Owner

stgraber commented Jun 23, 2016

Note that indeed, unless you have http_proxy and https_proxy set in the daemon's environment (which init systems don't usually do), you will need to set core.proxy_http and core.proxy_https

This likely was the first problem you ran into, and you then hit the actual image server being unavailable at the moment.

Okay, thanks, but what is the reason I could curl without a problem? I tried that for about 2 hours in different env combinations, and in all cases curl was working but lxd launch was not. Maybe it can be caused by proxy being of form http://abc:1234/, even though it is a https proxy?

Owner

stgraber commented Jun 23, 2016

And you had both core.http_proxy and core.https_proxy set to http://abc:1234 during those tests as well as http_proxy and https_proxy set accordingly in the shell you were running the lxc command from?

Well, now knowing that there are problems with cloud-images.ubuntu.com, I have tried

lxc launch images:ubuntu/trusty/i386 t2 --debug

and now it worked, but only if core.https_proxy was set, it ignored $HTTPS_PROXY, as you mentioned in your previous comment. However, I am not sure how to resolve this issue, because this is a controller machine bootstraped by Juju 2.0, especially that it requires doing

lxc profile unset default environment.http_proxy
lxc profile unset default user.network_mode

(or at least the latter one)

Owner

stgraber commented Jun 23, 2016

Both of the profile configuration keys should automatically get unset once lxdbr0 is properly configured which I'd expect Juju to do for you.

Same should go for core.proxy_http and core.proxy_https, those should be set by Juju when it's told to use a proxy for a given environment.

As for the environment variables, LXD doesn't ignore them, but they need to be set for both the client (lxc) and the daemon (lxd). Init systems don't pass those environment variables so unless you manually modified the lxd systemd unit, the daemon wouldn't have them set.
Since having folks modify init scripts is a bit of a pain, that's why we introduced those daemon configuration keys.

Actually, in most cases you don't even need the client to have the environment variables set as it's the daemon doing all the image handling, not the client. The exception is for commands that are done directly by the client such as "lxc image list ubuntu:" or "lxc image info ubuntu:16.04".

Owner

stgraber commented Jun 23, 2016

Oh and yes, cloud-images.ubuntu.com is now back online.

ptylenda commented Jun 23, 2016

In case of juju, this is how I am bootstrapping.

config.yml:

apt-http-proxy: http://some_proxy:911/
apt-https-proxy: http://some_proxy:912/
bootstrap-timeout: 3600
enable-os-refresh-update: true
enable-os-upgrade: true 
http-proxy: http://some_proxy:911/
https-proxy: http://some_proxy:912/
image-stream: daily
no-proxy: [some ips]

Command:
juju bootstrap kymani maas/http://172.28.188.200/MAAS --config ~/juju_bootstrap_configs/kymani-maas.yaml --upload-tools --bootstrap-series=xenial --credential=kymani --show-log --debug

Such a controller gets bootstrapped properly, I am able to communicate from this machine to outside world, use apt-get and so on, but lxd is not configured properly out of the box. Well, it may be a bug in beta version...

UPDATE: I might have not been precise enough, I am bootstraping maas controller on juju in order to deploy openstack. And I have tried to use different bundles, but these with lxc are not compatible with juju 2.0, because they need to use "lxd" instead of "lxc" in configuration files. The problems I describe occur when I use config https://api.jujucharms.com/charmstore/v5/~openstack-charmers-next/openstack-base-xenial-mitaka/archive/bundle.yaml with all lxc changed to lxd, as described in https://jujucharms.com/docs/devel/temp-release-notes (https://jujucharms.com/docs/devel/temp-release-notes#lxd-containers)

mushuweb commented Aug 12, 2016

Proxy configuration

In most setups, you’ll want the LXD daemon to fetch images from remote servers.

If you are in an environment where you must go through a HTTP(s) proxy to reach the outside world, you’ll want to set a few configuration keys or alternatively make sure that the standard PROXY environment variables are set in the daemon’s environment.

lxc config set core.proxy_http http://squid01.internal:3128
lxc config set core.proxy_https http://squid01.internal:3128
lxc config set core.proxy_ignore_hosts image-server.local

With those, all transfers initiated by LXD will use the squid01.internal HTTP proxy, except for traffic to the server at image-server.local

Source : https://www.stgraber.org/2016/03/15/lxd-2-0-installing-and-configuring-lxd-212/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment