Is it possible to start multiple instances of VMs at the same time?

[antranapp]

We have a bunch of Mac Studios for CI in our office at this moment but we can use it only for run a single job at a time right now due to flaky Xcode with parallel executions.

Can Cilicon start multiple VMs at the same time, which run independent from each others? This'd help us to ultilise the Mac Studios' resources better by parallelising multiple jobs in multiple VMS

[Marcocanc]

Hi @antranapp, that's definitely something we're considering to implement, however I can't give a clear timeline. In theory it should be possible to run up to 2 VMs in parallel. While I haven't tried it myself I've heard from several people that 2 VMs is a limitation of the Virtualization framework.

[AaronBurchfield]

While it may not be intentional, this works already by starting a second instance of Cilicon. It would be awesome if a single instance were able to manage multiple vms with different configurations to support building on different macos versions.

[Sherlouk]

I've heard from several people that 2 VMs is a limitation of the Virtualization framework

It's not a limitation of the framework or the hardware, but rather the legal agreement. You can find it here and the important part is under 2.b.iii:

"to run up to two (2) additional copies or instances of the Apple
Software within virtual operating system environments on each Apple-branded computer" (shortened for brevity)

You will get a VZErrorDomain error 6 if you try to add 3 or more which tells you it's not allowed. This is a very old part of the agreement though, so hopefully if enough people ask Apple to amend it they may open it up on Apple Silicon to allow more since clearly with how powerful the new chips are it can definitely support it.

---

With that said, I'm adding +1 to the ticket as I'd love to see support for two added for the reason above of wanting to support multiple CI test jobs in parallel without having them impact one another.

[ivan-gaydamakin]

Any updates?

[ast3150]

This would be very interesting for us. What's needed in terms of implementation to support this feature?

[ast3150]

Fyi opening Cilicon multiple times (using open -n -a Cilicon) results in the network connection being dropped for both instances of Cilicon

[Halle]

Hiya @ast3150 and @Marcocanc , I am running Cilicon right now with two VMs and it seems to be working very well (Cilicon is great full-stop, thank you @Marcocanc for publishing and supporting this project – it really hits the right balance for me between believably ephemeral instances and low-fussiness systems complexity, given a moderate need). I am doing multiple VMs via a pretty easy workaround, but it is my hope to find some discretionary time so I can submit a real PR for this feature (and possibly a couple of other itch-scratches I'm doing locally). For now I will just document the workaround, which is having two running app instances of Cilicon where the second one looks for a config entitled cilicon2.yml (this requires building your own Cilicon.app for the second app/instance).

1. Open Cilicon.xcodeproj and change every reference to the build target Cilicon in the project and its build settings to Cilicon2 so you can export an archive build to /Applications to run alongside Cilicon.app. I also changed the PRODUCT_BUNDLE_IDENTIFIER and used my own codesigning in order to separate other potentially shared resources I don't know about that use the bundle domain. It is entirely possible that this is unnecessary and it's enough to simply rename the built app, but I decided to do it this way in order to avoid wondering about edge cases when I had other mysteries to debug.

2. Change the ConfigManager.swift line static let configPaths = ["/cilicon.yml", "/.cilicon.yml"] to static let configPaths = ["/cilicon2.yml", "/.cilicon2.yml"].

3. Build app, put Cilicon2.app product wherever you put Cilicon.app.

That's actually it, then you can launch both apps and at least for me, they are both getting network and can both launch different correctly-configured VMs that appear to work correctly so far.

[Halle]

I'm guessing that the design of this inside a single app could be something as low-key as "if a cilicon2.yml exists, open a second window, do all the stuff" and you could probably leave it to the enduser to deal with the problem of providing a good cilicon2.yml, at least to start.

[Marcocanc]

Hi Halle, I will start working on a (much-needed) refactor of Cilicon with a much cleaner architecture and support for multiple VMs soon. Let me know if you'd like to contribute and we can try find a way to collaborate on it.

[Halle]

I'd enjoy that!

[Halle]

BTW, there is one other change needed in code to support this currently in the form I described above. If you follow my instructions above, you will eventually encounter this issue where a complete run leads to a permanent shutdown of one or the other instance. I believe the reason for this is that in the current logic, there is a brief period in which there would be three instances of virtualized macOS, which isn't allowed. I changed this in setupAndRunVirtualMachine():

        Task { @MainActor in
            vmState = .running(virtualMachine)
            try await virtualMachine.start()
        }

to first check for whether there is a VM in the .running state in the instance, and if so, to first stop it before starting it, and also to set vmState to the .running state after virtualMachine.start() is done waiting instead of before. This results in a slow restart after there has been a runner run (maybe there is a race condition?), but it is successful.