How to access CRC or apps deployed on CRC remotely

[morningspace]

General information

• OS: Linux
• Hypervisor: KVM
• Did you run crc setup before starting it (Yes)?

CRC version

# Put the output of `crc version`
version: 1.0.0-beta.5+f2aa58c
OpenShift version: 4.1.14 (embedded in binary)

CRC status

# Put the output of `crc status`
CRC VM:          Running
OpenShift:       Running (v4.x)
Disk Usage:      13.68GB of 16.09GB (Inside the CRC VM)
Cache Usage:     14.99GB
Cache Directory: /home/morningspace/.crc/cache

Host Operating System

# Put the output of `cat /etc/os-release` in case of Linux
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
... ...

Steps to reproduce

After I deploy CRC on a linux server running in LAN. How can I access CRC web console, or apps deployed onto it (if any), from another machine in the same network?

It looks the current CRC is designed to be used on local machine. So, it has ways such as Network Manager for Linux or /etc/hosts + /etc/resolver/testing for Mac to route requests outside to OpenShift running inside a VM. But it seems all happen on the same machine.

On the other hand, the original oc cluster up has the ability to support cross-machine access, because it runs on the machine directly (w/o VM) and it depends on the simple public DNS service nip.io(by default).

This seems to be the missing part in CRC? Or, if there's anything I missed.

I'd think this is a common usage scenario for dev/test/demo purpose at team level.

Expected

I can access CRC or apps deployed on CRC from another machine

Actual

I can only access it on the machine which is deployed CRC.

[gbraad]

I can only access it on the machine which is deployed CRC.

This was the same for Minishift unless you used the 'generic driver' to deploy this without a VM. This is by design as your are running this on a virtual network. The intended use-case is to use this for development and any unauthorized reote access would not be a wanted situation.

However, there are ways around this, but they are for now outside the scope of CRC. You could for instance allow the use of ssh with a proxy. Or, depending on the platform, reconfigure the network to use a remotely accessible network segment.

[morningspace]

@gbraad What does “the 'generic driver' to deploy this without a VM” mean? Is that also supported by crc? It looks I couldn't find that from crc docs.

Also, I found this issue on Minishift which is similar and you are in the loop as well :-)

It looks Minishift supports --public-hostname and --routing-suffix, just like oc cluster up that natively runs on host machine w/o VM, so that it can use 127.0.0.1.nip.io with ssh port forwarding to resolve the issue. Not ideal, but quite simple.

However, it looks crc does not has similar feature yet, the host/domain name seems to be fixed (crc.testing, apps-crc.testing) and not changeable. Please correct me if I'm wrong.

[gbraad]

Is that also supported by crc

No, not supported by CRC as we need to run on RHEL CoreOS, and therefore a VMM is needed.

We are currently getting ready for the GA and after this will continue with looking into some of the networking issue related to CRC, like proxy, etc.

We have purposely made very opinionated choices, very different from Minishift, that guarantees a consistent deployment and predictable cluster deployment, and therefore you are not able to modify some of the names and domains. This also prevents people consider this a production-ready setup and deploy CRC as a headless environment, like a server. Sure, they can, but the constraints are the same. The way aroudn this would be to use the snc repo, but at the moment the customization experience is far from ideal since it needs at least 24hours to generate an image (to force the initial certificate rotation: #11).

[morningspace]

@gbraad
To have it run as headless server and access remotely would probably be the normal use case for us. So, before CRC provides official ways in its future releases, I was thinking about the quick workaround... After googled around, I think it seems the most possible approach is to launch additional proxy outside CRC VM, then connect to it.

Please correct me if any. The major problem is that, usually, we may have routes that are all mapped to the same IP, with just different subdomains. e.g.

$ oc get routes -n istio-system
NAMESPACE                  NAME                HOST/PORT                                                 PATH   SERVICES            PORT              TERMINATION            WILDCARD
istio-system               grafana             grafana-istio-system.apps-crc.testing                            grafana             http                                     None
istio-system               jaeger-query        jaeger-query-istio-system.apps-crc.testing                       jaeger-query        query-http                               None
istio-system               kiali               kiali-istio-system.apps-crc.testing                              kiali               http-kiali                               None
istio-system               prometheus          prometheus-istio-system.apps-crc.testing                         prometheus          http-prometheus                          None

They all point to the CRC VM IP.

With that, I couldn't use tricks such as IP forwarding or port forwarding, but instead may have to leverage reverse proxy which supports name-based forwarding, e.g. virtual host w/ Apache, or subdomain w/ nginx, and so on.

Any better ideas or suggestions?

[morningspace]

After investigated for a few hours, it looks to use proxy is also not very straightforward as I thought originally. Because of the requirement of name-based forwarding, I chose nginx as example:

First, it's easy to expose route for HTTP service, and configure proxy. Here's an example:

server {
    listen 80;
    server_name grafana-istio-system.192.168.10.100.nip.io;
    location / {
        proxy_pass http://grafana-istio-system.apps-crc.testing;
    }
}

I just forward the request from client to the host machine (192.168.10.100), then to the VM (apps-crc.testing).

The tricky part is HTTPS, typically, the OpenShift web console. Ideally, to keep it simple, I would have my reverse proxy running using SSL passthrough rather than SSL termination, so, run at TCP level:

stream {
    server {
        listen 443;
        proxy_pass apps-crc.testing:443;
    }
}

However, because lack of the HOST HTTP header, I cannot access the OpenShift web console unless I add the below mappings into my local /etc/hosts:

192.168.10.100    console-openshift-console.apps-crc.testing api.crc.testing oauth-openshift.apps-crc.testing

This is inconvenient. Otherwise, I have to use tricks such as ngx_stream_ssl_preread_module where it can extract hostname from data packet via SSL, then determine the right host to send to. However, usually this is not the built-in module distributed w/ nginx which makes things a bit more complicated.

Essentially, I'd think that's why we may still need --public-hostname and --routing-suffix similar to minishift or oc cluster up if we want to support this scenario in future.

[morningspace]

@gbraad

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[morningspace]

Any update? Will this be in the plan? @gbraad

[cfergeau]

This is definitely not one of the supported usecases for crc. However this request comes up regularly. We could look into making it easier to do this manually, but I don't think we want to support it out of the box.

[morningspace]

making it easier to do this manually, but I don't think we want to support it out of the

Understood and that should be greatly helpful if it can be made easier when do it manually. That's enough for me. And, I'd think that can also get more people to use CRC.

I'd be interested to be involved if there's somewhere on this under discussion now or later. Thanks!

[morningspace]

@cfergeau

[jeffsaremi]

This is our use case.
Is there a workaround that we could use? This is really important for us.
Thanks

Here's the problem we have:

based on what we were told we cannot install crc on ubuntu 16.04 because libvirt is old.
so we're trying the following configuration where the crc is installed on the developer's desktop (Windows) and on ubuntu the person can use oc client (which runs on ubuntu 16.04) to connect and do what is needed.
For that we copy kubeconfig from C:\users\user_name.crc\cache\crc_hyperv_4.3.0 to the ubuntu machine.
setup dns entries using the following:
$ cat /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf
[main]
dns=dnsmasq
$ cat /etc/NetworkManager/dnsmasq.d/crc.conf
server=/apps-crc.testing/10.91.90.161
server=/crc.testing/10.91.90.161

and then try connecting. We get this:

$ oc login -u kubeadmin -p '...'
error: dial tcp: lookup api.crc.testing on 10.193.8.10:53: no such host - verify you have provided the correct host and port and that the server is currently running.

[morningspace]

@jeffsaremi

Based on my CRC use experience, I'd suggest you need a proxy installed along w/ CRC instance on the same machine, in your case, that's the WIndows system. I usually deploy CRC on a remote virtual machine, then install nginx on that machine as a reverse proxy and config the proxy to expose the CRC local network to others in the same LAN.

My understanding is that, CRC uses NetworkManager for linux OS, or etc/hosts in MacOS, and probably Windows(I haven't tried on Windows) to config a local network w/ hostname, IP mapping, i.e. api.crc.testing <local_crc_ip>, this is the IP pointing to the CRC VM, not the host machine.

Another thing to note is, api.crc.testing is the fixed hostname you have to use when you connect to the remote CRC instance from other machine. Because of that, I previously add the mapping manually in etc/hosts on my MacOS, which is a bit inconvenience. But recently, I found that can be changed by creating additional route in OCP (that's running in CRC), which is cool.

Here's a simple flow to demonstrate how I did it:

[my MacOS] --<access via hostname of remote VM>--> [my remote VM]

And, on that remote VM:

[nginx on my remote VM] --<access via api.crc.testing>--> [CRC VM]

[jeffsaremi]

@morningspace
Thanks for he instructions. I think i have setup everything properly but still cant get the name resolved.
On the Windows Machine:

• nginx installed
• config:

stream {
  server {
    listen     6443;
    proxy_pass 10.91.90.161:6443;
  }
}

• crc ip:
  10.91.90.161
• ipconfig

Ethernet adapter vEthernet (crc):

   ...
   IPv4 Address. . . . . . . . . . . : 10.91.90.45
   Subnet Mask . . . . . . . . . . . : 255.255.252.0
   Default Gateway . . . . . . . . . : fe80::200:5eff:fe00:20d%22
                                       10.91.88.1

On the ubuntu16 machine:

- sudo apt install dnsmasq
jesaremi@u16-2:~$ cat /etc/NetworkManager/conf.d/
10-ubuntu-fan.conf              crc-nm-dnsmasq.conf             default-wifi-powersave-on.conf
jesaremi@u16-2:~$ cat /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf
[main]
dns=dnsmasq
jesaremi@u16-2:~$ cat /etc/NetworkManager/dnsmasq.d/crc.conf
server=/apps-crc.testing/10.91.90.45
server=/crc.testing/10.91.90.45

(not the IP address is the one for the Windows and not oc vm)

• sudo systemctl restart NetworkManager

$ nslookup api.crc.testing
Server:         127.0.0.1
Address:        127.0.0.1#53

** server can't find api.crc.testing: NXDOMAIN

jesaremi@u16-2:~$ nslookup crc.testing
Server:         127.0.0.1
Address:        127.0.0.1#53

** server can't find crc.testing: NXDOMAIN

• $ telnet 10.91.90.45 6443

Trying 10.91.90.45...
Connected to 10.91.90.45.
Escape character is '^]'.

$ grep dns /var/log/syslog

Mar  5 12:41:57 u16-2 systemd[1]: Started dnsmasq - A lightweight DHCP and caching DNS server.
Mar  5 12:43:43 u16-2 dnsmasq[76368]: exiting on receipt of SIGTERM
Mar  5 12:43:43 u16-2 NetworkManager[1125]: <info>  [1583441023.3005] dns-mgr: init: dns=dnsmasq, rc-manager=resolvconf, plugin=dnsmasq
Mar  5 12:43:43 u16-2 NetworkManager[1125]: <info>  [1583441023.3016] dns-plugin[0x7fab50008e90]: starting dnsmasq...
Mar  5 12:43:43 u16-2 NetworkManager[1125]: <info>  [1583441023.3069] dns-mgr: Writing DNS information to /sbin/resolvconf
Mar  5 12:43:43 u16-2 dnsmasq[78907]: started, version 2.75 cache disabled
Mar  5 12:43:43 u16-2 dnsmasq[78907]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC loop-detect inotify
Mar  5 12:43:43 u16-2 dnsmasq[78907]: DBus support enabled: connected to system bus
Mar  5 12:43:43 u16-2 dnsmasq[78907]: using nameserver 10.91.90.45#53 for domain crc.testing
Mar  5 12:43:43 u16-2 dnsmasq[78907]: using nameserver 10.91.90.45#53 for domain apps-crc.testing
Mar  5 12:43:43 u16-2 NetworkManager[1125]: <info>  [1583441023.3307] dnsmasq[0x7fab50008e90]: dnsmasq appeared as :1.596
Mar  5 12:43:43 u16-2 dnsmasq[78907]: setting upstream servers from DBus
Mar  5 12:43:43 u16-2 dnsmasq[78907]: using nameserver 10.91.90.45#53 for domain crc.testing
Mar  5 12:43:43 u16-2 dnsmasq[78907]: using nameserver 10.91.90.45#53 for domain apps-crc.testing
Mar  5 17:41:35 u16-2 NetworkManager[218259]: <info>  [1583458895.3138] Read config: /etc/NetworkManager/NetworkManager.conf (etc: 10-ubuntu-fan.conf, crc-nm-dnsmasq.conf, default-wifi-powersave-on.conf)
Mar  5 17:41:35 u16-2 NetworkManager[218259]: <info>  [1583458895.3272] dns-mgr[0x215b950]: init: dns=dnsmasq, rc-manager=resolvconf, plugin=dnsmasq

[jeffsaremi]

@morningspace
Further to my notes above, if I just add

$ cat /etc/hosts
127.0.0.1       localhost
ff02::2 ip6-allrouters
<crc ip> console-openshift-console.apps-crc.testing api.crc.testing oauth-openshift.apps-crc.testing

it works perfectly. I don't need the nginx this way!

[jeffsaremi]

@morningspace
Leaving yet another update:

$ tail /etc/dnsmasq.conf

no-hosts
addn-hosts=/usr/local/etc/hosts

and in /usr/local/etc/hosts I added:
<crc ip> console-openshift-console.apps-crc.testing api.crc.testing oauth-openshift.apps-crc.testing

NOTE: no nginx needed anywhere

$ sudo /etc/init.d/dnsmasq restart

$ oc login -u kubeadmin -p 7z6T5-qmTth-oxaoD-p3xQF https://api.crc.testing:6443
Login successful.

You have access to 54 projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "default".

[morningspace]

Interestingly, @jeffsaremi trying to understand what you did... so:

In this comment, I'm assuming you are working on your Ubuntu machine, and what's the <crc ip> stands for, is it the Windows machine IP, or the CRC VM IP?

In this comment, it's also the Ubuntu machine right? And you are using NetworkManager w/ dnsmasq plugin, that points to the /usr/loca/etc/hosts, w/ the manually added host, IP mapping, right?

Lastly, I'd like to know how you configured on your Windows machine. Since CRC is a VM sitting inside Windows as its host machine, how the request can go into the VM via the host machine? (I haven't tried CRC deployment on Windows yet)

[jeffsaremi]

@morningspace
Indeed I owe some explanation here.
Firstly on windows prior to installation, I created a Virtual Switch with the name 'crc' which got picked up by 'crc setup'
This switch was of type external meaning that all IP addresses were actually externally accessible.
That's why I guess I didn't need the proxy anymore.
'crc ip' gives the OC VM's Ip address . So the actual Windows IP was never needed here.

I tried accessing this from a ubuntu 16.04 machine with the oc command line.
As you can see in my initial attempts I tried setting up dnsmasq from NetworkManager.
This never worked for me.

So I decided to configure dnsmasq with the hosts file option. And the line that you had mentioned in the hosts file worked perfectly for me with just the OC vm ip address.

[morningspace]

@jeffsaremi
This is awesome! Now I understand how the magic happens :-) So, the reason that I need a reverse proxy is just like you need Virtual Switch, in order to expose the CRC VM IP.

The only thing remained that I wonder is how people deal w/ multiple CRC instances running in the same network. Have you tried to use a different hostname other than api.crc.testing in your /etc/hosts? IIRC, this can be arbitrary hostname as long as that can be resolved by DNS and properly goes to CRC VM.

But for the web console, what I learned is that I need to create OCP route, otherwise, I will have to create multiple IPs pointing to the same sort of hostnames, console-openshift-console.apps-crc.testing, oauth-openshift.apps-crc.testing, in /etc/hosts and switch among them manually if I want to access multiple CRC instances in different time.

[jeffsaremi]

@morningspace
I did not try different names however in one of my attempts I replaced api.crc.testing with just an ip address. Then upon oc login you'd get prompted if you wanted to accept this exception (being the certificate name and the host name mismatch). But then got stopped at the next name resolution.

$ oc login -u kubeadmin -p '<password>'
The server is using a certificate that does not match its hostname: x509: certificate is valid for 172.30.0.1, not 10.91.90.161
You can bypass the certificate check, but any data you send to the server could be intercepted by others.
Use insecure connections? (y/n): y
error: dial tcp: lookup oauth-openshift.apps-crc.testing on 10.193.8.10:53: no such host

I have not used oc route and didn't know this existed. If you have more instructions let me know and I'll try.

thanks

[morningspace]

@jeffsaremi
So, that appears to be the DNS lookup issue, rather than the certification issue.

I tried curl apiserver using hostname other than api.crc.testing, it can work w/o any additional change:

curl -kL https://<your_host_running_crc>:6443

But for console-openshift-console.apps-crc.testing and oauth-openshift.apps-crc.testing, it requires DNS and additional routes, e.g. here're the two default ones:

$ oc get routes --all-namespaces
NAMESPACE                  NAME                                               HOST/PORT                                                 PATH   SERVICES             PORT    TERMINATION            WILDCARD
openshift-authentication   oauth-openshift                                    oauth-openshift.apps-crc.testing                                 oauth-openshift      6443    passthrough/Redirect   None
openshift-console          console                                            console-openshift-console.apps-crc.testing                       console              https   reencrypt/Redirect     None

Just clone them and create your own by using your own hostname, it will work.

[jeffsaremi]

@morningspace
Thanks a lot for this great info. I'll apply once I get a chance

[ahaerpfer]

For what it's worth … I just came across this Red Hat blog entry that might also help:
Accessing CodeReady Containers on a Remote Server

[jeffsaremi]

@ahaerpfer thanks for the article. Such elaborate set steps! It looks like we need another crc-like program just to access crc!
I like the step of adding directly to the /etc/hosts file. The fastest way to get there.
Also I don't understand why we can't just use the IP address when issuing an oc command?
That would even eliminate the need for /etc/hosts or Networkmanager modifications.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[gbraad]

@tmckayus would you be able to assist @kowen in getting this document properly? (In reference to: https://gist.github.com/tmckayus/8e843f90c44ac841d0673434c7de0c6a)

[tmckayus]

@tmckayus would you be able to assist @kowen in getting this document properly? (In reference to: https://gist.github.com/tmckayus/8e843f90c44ac841d0673434c7de0c6a)

@gbraad sure I can help. Not sure what we've got in mind.

[tmckayus]

@robin-owen ping on this ^^, is there something I can help with?

[kowen-rh]

@tmckayus Hi there! Apologies for the late response. I've been working on getting this particular story documented, but have no way to verify the steps that I've been documenting -- as such, we can't reasonably include this in the docs yet. Would you mind taking our conversation to email so that we can go through the steps you've outlined in your gist and convert that into documentation?

[tmckayus]

for history, taking to email :)

@tmckayus Hi there! Apologies for the late response. I've been working on getting this particular story documented, but have no way to verify the steps that I've been documenting -- as such, we can't reasonably include this in the docs yet. Would you mind taking our conversation to email so that we can go through the steps you've outlined in your gist and convert that into documentation?

[devalru]

Accessing CodeReady Containers on a Remote Server

Not working on CentOS 7 and haproxy-1.5.18-9.el7.x86_64.
Fix: Adding port 443 to line "server webserver1 CRC_IP:443 check" (https://gist.github.com/tmckayus/8e843f90c44ac841d0673434c7de0c6a#gistcomment-3228253)

[kowen-rh]

This has been fixed for some time now by #1473 and #1515. Closing this issue.

Thank you, everyone!

[imperialguy]

But recently, I found that can be changed by creating additional route in OCP (that's running in CRC), which is cool.

@morningspace How can apps-crc.testing be changed? Say, I want the URL to be openshift.example.com, how do I do that? Assuming openshift.example.com is already pointing to the machine that is running crc.

[cfergeau]

@imperialguy crc's wiki has https://github.com/crc-org/crc/wiki/Change-the-domain-for-CRC but I don't know if these instructions are still current.

[TheiLLeniumStudios]

I wrapped everything around an automated script to setup everything. Feel free to check it out: https://github.com/iLLeniumStudios/remote-crc-setup

Aiming to make it more configurable via vars soon