Curl hang on querying VM IP

[eecsmap]

Recently I am working on a service heavily relying on the API /vms/{vmname}/ip. Yet I have run into couple of times that the query simply hangs.

user@host-01 ~ % oc -v  https://orchard.mycompany.net/v1/vms/testvm/ip
* Host orchard.mycompany.net:443 was resolved.
* IPv6: (none)
* IPv4: 10.0.0.11
*   Trying 10.0.0.11:443...
* Connected to orchard.mycompany.net (10.0.0.11) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: C=US; ST=California; L=Palo Alto; O=MyCompany Inc.; CN=*.mycompany.net
*  start date: Apr 10 00:00:00 2024 GMT
*  expire date: Apr 10 23:59:59 2025 GMT
*  subjectAltName: host "orchard.mycompany.net" matched cert's "*.mycompany.net"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
* using HTTP/1.x
* Server auth using Basic with user 'orchard-root'
> GET /v1/vms/testvm/ip HTTP/1.1
> Host: orchard.mycompany.net
> Authorization: Basic xxxxxxxxxxxxxxxxxxx
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off

Then it hangs.

This issue is not observed on my prod environment. Yet happens in my test environment with a large chance. Any suggestion?

[eecsmap]

It seems hang on waiting for https://github.com/cirruslabs/orchard/blob/main/internal/controller/api_vms_ip.go#L58

[edigaryev]

Hello!

1. Are these hangs reproducible for a short period of time after a single hang occurs, or they're completely stray with only a single one occurring in a minute/hour/day timeframe?

2. Are you using --net-bridged? Using --net-bridged causes Orchard to pass --resolver=arp to the tart ip command, which in turn relies on the VM being active on the local network to populate the host's ARP table.

   Most Linux VMs don't do that out-of-the box and this happens sometimes with long-running macOS VMs.

3. The next time you observe this, could you please verify that tart ip orchard-<VM-name>-<UUID> (or tart ip --resolver=arp orchard-<VM-name>-<UUID> when using --net-bridged) works on the node with your VM at the time this happens?

[eecsmap]

When I run tart ip <vmname> I got no IP address found
When I run tart ip <vmname> --resolver=arp I also got no IP address found

I double checked with the DHCP server and it actually allocate the IP for the MAC of the VM. I can ssh into the VM using this IP.

[eecsmap]

I think I got the problem fixed. Thanks!

[Formartha]

I'm having the same issue,
when running a machine using orchard, I'm able to see that the worker opened the machine, I can manually (using tart) see it's internals but I can't resolve it's IP address.

I'm using bridged mode as it needs to communicate with external systems.

What have you done to resolve it?
p.s. it does work with resolver --arp

[edigaryev]

@Formartha please see Resolving the VM's IP when using bridged networking.

Also related to cirruslabs/tart#1020.

[Formartha]

Gothca. @edigaryev
One more Q: can we have more than one VM running at the same host machine which uses bridged adapter on the same interface?

[edigaryev]

One more Q: can we have more than one VM running at the same host machine which uses bridged adapter on the same interface?

Yes, you can.

[Formartha]

Superb, it works.
I still am unable to understand how to solve the ip resolution even after installing samba on the orchard manager.
Does it require both to be in the same network segment?

My architecture is that the macs are stored locally and the manager is an ec2 instance running a container.

There is a route between the physical Mac and the remote ec2 instance, but its not the same network.

[edigaryev]

I still am unable to understand how to solve the ip resolution even after installing samba on the orchard manager.

Please see cirruslabs/tart#1020.

The problem is that a second VM gets ignored by the host's NBNS resolver because it has the same name as the first VM.

Would you mind explaining the reason for using the --net-bridged in the first place? Perhaps there are other ways to accomplish this?

[Formartha]

Thanks @edigaryev

We have a product which uses orchestrator and agents. The orchestrator is an instance (service) in AWS and it connects to a local machine and instals an agent on. The remote orchestrator and the agent share data between them and the connection is being done two ways.

Without bridge I won't be able to do it, as far as I tried.

Also,
to be able to install the agent, we'd need (as a part of deployment flow) to access the VM and run a set of commands which will fetch and deploy the agent on the newly created VM. This is a part of a CI process I'm aiming to introduce.

[edigaryev]

Just to clarify: do you only need the ability to e.g. SSH to the VM to configure the agent?

Or your orchestrator also needs to be able to reach agent's TCP/UDP port on that VM?

[Formartha]

I’d need both.

1. Installing the agent will require a ssh connection
2. Using the orchestrator and the agent in a call flow will require TCP connection between them

[edigaryev]

will require TCP connection between them

And the initiator will be the controller? Or the agent? Or both?

Note that in case it'll be the agent who initiates the connect, you don't need bridged networking as it should work just fine with shared (NAT) networking.