Linux emulator fails on macOS M1 even with x64 virtualization #54
Comments
|
Please do something for start it on Mac M1 .. |
|
This would be an awesome addition since we are seeing more and more M1's as dev boxes. |
|
I would also appreciate seeing this working. 🙏 |
|
2+ years past since WWDC 2020. We're all need support for Apple silicon. |
|
100% agree that the emulator should work on arm64. I've created this azure feedback post on the issue. Don't know if Microsoft care about it, but worth a shot. |
|
FYI we ended up migrating off CosmosDB because of this |
Don't go down that route! |
|
Try asking here https://learn.microsoft.com/en-us/answers/topics/azure-cosmos-db.html |
|
maybe @jongio know who to contact or existing workaround / replacement |
|
@markjbrown would likely know |
|
There's no workaround for this unfortunately. This is on our roadmap to deliver but I don't have any ETA. |
|
can you point out where the source is for the MSI hosted at https://aka.ms/cosmosdb-emulator that is in the Dockerfile maybe someone might find out how to get update (replace/remove) all the servercore specifics to make it more XPlat ? |
|
The emulator is not open source. |
|
should it be concidered to open source it ? if not, should all the communication protocol be open / documented or is it already ? private impl could be kept private It can only benefits everyone |
|
Yes, I will definitely take that suggestion to the team. Thank you! |
|
Please do whatever it takes to prioritise and get this resolved soon. Not having proper tool support is a show stopper. Everything works perfectly in my development environment on the MacBook Pro M1, except the Azure Cosmos DB Emulator. My current workaround is to run the Azure Cosmos DB Emulator on Parallels Desktop, but that workaround has a lot of disadvantages, and I wouldn't need Parallel Desktop if it wasn't for this... |
|
Please make sure to make it work for MacBook Pro M1. That will be a BIG help. |
|
Would love to see M1 support. |
|
Not having M1 support is a big limitation, and it has definitely a major impact when it comes down to choosing a Cloud platform and/or Azure resource selection. |
|
Agreed. Between this and not supporting mssql on aarch64 in docker, I'm concerned about proper development support of these products. Seems they're trying to rely on Docker supporting Rosetta (which is supposed to be a temporary/transition piece, not a something to rely on long term). |
|
Oof, ran into this issue today. We need this for our Cosmos db app development as well. |
Unfortunately the people responsible of this product doesn’t seem to care that much about this? Everything works just fine for me as .NET developer on a MacBook Pro M1 except the Azure Cosmos DB Emulator. |
Yep, same for me as dotnet/react developer. CosmosDB emulator the only one software on the world, that not runs at all on my arm64 Mac. It's looks insane for 2023. It not works even in latest Docker "Rosetta 2 in Linux" feature. |
|
The latest beta of Docker Desktop has a new feature for better support of running x86/64 binaries with Rosetta 2. Would this make it possible to run the Linux Emulator on M1 chips? I haven’t had the possibility to test this my self yet, so if anyone wants, please try it out and reach back here with the result please :) More info about this can be found here: |
Nope. I've tested it already. Emulator Just run & die immediately after start. |
|
Hi folks. Thanks for all your comments. As I mentioned back in December this is on our roadmap, but I don't have an ETA. It might help if I explain how features get committed on our team. I think most folks here understand this, but I want to take a moment to explain to help set expectations and provide some solace to those who may not think we care or paying attention to this. The Cosmos team (along with every service team in Azure) organizes feature work in 6-month semester that starts after a month of planning where work items are committed. The current semester started in October and runs through March. Despite Microsoft's size, feature teams are very lean. Items that are committed are done after careful resource planning to ensure we don't over commit, resulting in slips or dropped items. This means that even if we see intense demand for something after planning, (like this one here where the bulk of input occurred after the semester had started), it is unlikely we could take it on. This would often require dropping other committed work which could lead to a cascade of issues for downstream dependencies. The planning for our next semester (April-October) will start sometime near the end of February next month. M1 support is among the list of features that will be reviewed for this upcoming semester. This GitHub issue here that you have been commenting and voting the feature up on, has been filed in support of the feature and is being followed. We recognize and appreciate your continued interest and support for this. For people who are seeing this Issue for the first time, you are welcome to add your support as well. Once planning for our upcoming semester has concluded, myself or another on our team will revert back here on the outcomes. Thanks again. |
|
@markjbrown thanks for the insights on feature semesters. Don't sweat it - I'm just hoping for this to get committed to in the next semester, fingers crossed 🤞 And trying to create some buzz here to get the necessary attention 😉 |
|
Would really love to see this. Tis a big deal for integration tests and regular development flows. |
|
Given the 2023 Developer's Survey from StackOverflow indicates 33% of professional developers use MacOS, support for the newer Apple silicon should be a much higher priority for Azure tooling. I've encountered so many Azure tools that lack this support, it's infuriating developing in this ecosystem. |
|
So after realizing I couldn't run Microsoft Server in VMWare Fusion on my M2, I started down the Cosmos Docker path and wound up here. It's almost like Microsoft and/or Apple don't really want to developers to work cross platform. How odd. |
|
Any update on this? |
|
Apple has released the m3 lineup, but we are still without the support of Apple Silicon here, even though docker x86_64 translation to arm64 via Rosetta 2. Sad, just very sad, it shows us how Microsoft supports developers who have tried to invest in Azure cloud 😢 |
Use the SSH port-forwarding tunnel, it works fine. You only need install OpenSSH Server from Optional Features on Windows, run the service and then create the tunnel from Mac:
Don't forget change the network connection as Bridge. Note: When you enter to 127.0.0.1:8081/_explorer/index.html... download the certificate and install it, if not your Chrome or the default browser will show up a warning. |
|
Just a pointer for anyone attempting to use the VM workaround - I discovered today that version 2.14.12 appears to not bind to an open network interface anymore, and only localhost. I don't know why this is the case, but it very well could be the cause of people's connectivity issues here. It certainly was mine! I downgraded to 2.14.9 by finding a link to the package in the Chocolatey repo, uninstalling the latest version and installing 2.14.9 instead. Here's a direct link to the MSI installer. This resolved my connectivity issues immediately. So that may help people get unstuck here too. |
|
I'm doing development for azure and as apart of that I'm using Cosmos. I just got my bill and with only one database and a few containers I'm up to $200 for the month -- just for testing! Outrageous prices! I have to emulate Cosmos for local development or I'll need to use something like Mongodb. I'm on a M1 mac. So the steps to get this working as I've seen it here are (since docker is not an option):
Is this it? What am I missing? |
Sorry @aadamsx but this is not working anymore, I don't know why, maybe Windows's firewall rule? I tried to fix in different ways without exit, the database connect successfully but I can't save data on it. Well, at the moment I'm working under Windows using Parallels and works fine. |
Thanks. And you say you got this working by working within Windows? So you have to set up your azure functions or whatever functions are interacting with Cosmos within windows for this to work? |
|
Looks like I'm going to have to install Mongo: brew install mongodb-community@7.0 and work with this as an approximation of cosmos. Is this what everyone is doing to get around this? Of course the queries are different if you use the NoSQL interface for Cosmos instead of Mongo: https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/tutorial-query |
Most of us have just moved on to other databases at this point... |
Exactly, I'm working with Parallels for now using Visual Studio Community, but also I'm thinking to migrate to mongodb. |
|
Okay, I spun up a mongodb instance on railway.app ($5 a month plan to start). Migrated my data over from Cosmos to the new Mongo instance. Changed the connection strings on my Azure Functions, and everything seems to be working fine -- seemless and smooth transition so far. Once they figure this Arm64 transition for Cosmos, I'm willing to migrate back. |
|
@markjbrown I suspect that the April-October semester is most likely over, but this issue also did not make its way to the current one? |
|
Hi folks. First, I want to apologize, again. We know this is not a great experience for you. Also, apologies for not responding here sooner. I needed some time to gather information on the current state of all this so I could provide a more concise and accurate update. As I indicated over a year ago, we were looking to add this to our roadmap. I'd like to share an update on that work and also share a bit on the internals of our emulator to provide some transparency on what's involved. Our emulator is an actual mini-Cosmos DB, minus things that don't make sense on your personal machine, such as authentication, replication, etc. The Azure Cosmos DB code base is written in C++ with many dependencies using Windows-native system calls. For the past year as we have been working on porting it, we’ve been able to emulate many Windows-native system calls on Linux using the same PAL (Platform Abstraction Layer) that powers SQL Server on Linux. Unfortunately, PAL does not solve Apple Silicon compatibility. We’ve investigated multiple paths including using Rosetta for x86/amd64 emulation – and what we’ve found is that the effort to get all of Cosmos DB’s dependent Windows-native system calls working is high. This is why things are taking so long. While we continue to work on this, for some of you, we may have an alternative that might help
We have been following the guidance and help you have been giving each other. I love seeing developers helping each other. It makes me proud to be a part of this community and to be associated with you all. Thank you for sharing with each other. We want to make it easier for other developers to find this knowledge you've shared and share some other guidance we’ve learned. We are working on publishing this guidance you’ve shared as docs and samples on MS Learn to help others and will publish this soon. I again want to apologize again to those of you who have struggled due to our missing emulator support. We pride ourselves in being a developer-first database with an on our SDK's and developer experience. An emulator is a HUGE part of that experience and we have not lived up to our expectations. We recognize the need to do better and are committed to doing it. |
Why doesn't microsoft develop these products without depending on windows native functionality first? Seems like a no-brainer |
|
@markjbrown thanks for the update. I'm currently working on a project for a Microsoft Azure customer. And on my team, I'm having to use a serverless instance of Cosmos DB vs the emulator precisely because of this issue. This is a bit frustrating as it means I can't develop offline. But more to the point, I just ran into a scenario where I was unable to run integration tests locally because of a configuration change in the code for cosmos re throughput handling, which worked fine in the context of the emulator, but failed when using a serverless instance for the reasons re "shared throughput database creation is not supported for serverless accounts" outlined here. I appreciate the update and the suggested workarounds, but unfortunately it's still not the same as having actual parity in tooling with the rest of my team. Perhaps I missed it in the update, and I'm sorry if I have, but is there a commitment in terms of timing or planning on the roadmap for resolving this issue? I saw a bunch of workarounds (all using the cloud) and an explanation of why it's difficult, but I saw no information about which semester this is being slated to be tackled. Is there really a commitment here by Microsoft to make this work or is this just going to languish for more years to come? (And I'm sorry to say years, but at this point my M1 Macbook Air is now 3 years old... the M-series Macs aren't exactly new.) |
|
hi @nvanexan sorry for the slow reply. catching up. Serverless accounts do not have any throughput provisioned for them so anything you try to do with a serverless account and throughput will not work. I'm not sure why it appears to work in the emulator as it should not work at all. I don't have a date to share at this point. This work has been ongoing. I realize this has been a years' long issue. This issue did span COVID. This isn't a major reason for the length of time but for sure didn't help. When I feel like we have something we can share with you all here I will absolutely share it. PS: Make sure you understand our recommendations with shared database throughput. If you have containers with uneven requests and storage needs, this may not be a good choice to use across every container. Containers with shared database throughput are ok when each of the containers has roughly equal throughput and storage needs. Use dedicated throughput in the same database for containers which have greater needs. This will ensure that those containers scale with predictable performance. |
|
@markjbrown Is this likely to land this year? |
|
This the only thing that preventing me from completely switching my workstation to a mac. I appreciate the effort you guys are putting in. Hope the issue will be resolved sometimes soon |
Cynical thought, maybe they don't want people switching to the mac rofl. |
|
Choose automated tests. Choose an OS and machine that makes you productive. Choose reliability. Choose being able to work offline. Choose fast feedback. Choose products that support you. |
|
We're forced to use an actual Cosmos DB to perform integration testing on CI. The emulator is too slow, unlike DynamoDB local... An extra blob is also used as a mutex to guarantee there is only single CI job involved. |
|
I'm very appreciative of this tool and would like to use it on my Mac. Is there a timeline for this issue yet? Thanks again! |
and others
Understandably, you do not support M1 chips and it is questionable if supporting M1 chips is worth prioritizing; however, somehow the container fails on M1 even when using x64 virtualization, as demonstrated in #17.
At a minimum, can we have the container work with x64 emulation?
The text was updated successfully, but these errors were encountered: