-
Notifications
You must be signed in to change notification settings - Fork 58
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Virtualized Time Zone in Windows Server Containers #15
Comments
I'd like to see this feature coming out as well. There are a number of use cases we want to be able to change the time zone. It would be ideal if we could do that through docker/AKS arguments, leaving the host's timezone untouched. |
I agree. The inability to set a container timezone independent from the host is a significant limitation. In scenarios where applications need different timezones, the only solution is to sort them onto corresponding hosts with matching timezones. This is not the most convenient workaround in platforms where time consistency across hosts and infrastructure is critical for logging and troubleshooting. |
Hi all, thank you for raising your concerns here. The fix for this is actively in development and I will keep you all in the loop once it is ready for release. We recognize the importance of this issue so we're working to get the solution out as fast as possible. Again if you have specific concerns that you would like to mention I would be happy to hear it. Always helps me to hear customer use cases. Feel free to reach out to me via win-containers@microsoft.com, or on Twitter https://twitter.com/brandon8675_309. |
This issue has been open for 30 days with no updates. |
Hello @brasmith-ms Do you have any updates on this issue? |
Hi, We are also interested in any updates & the progress of this issue as it might impact our design/plan for migrating legacy applications into AKS windows containers. Thanks! |
Hi, @brasmith-ms and @vrapolinario |
I am also eager to use this feature. Any update? |
Hi all, thank you for your patience! The feature work is in its final stages and we are currently preparing it for the servicing pipeline. Taking the holiday servicing gap into account, our aim is to get this feature out to the public for the February/March servicing timeframe. I will continue to keep this page updated with status as we go, and again feel free to reach out to me if you have specific concerns. Rest assured we are working on getting this out as soon as we possibly can, as we do recognize the degree of its importance. An additional note, if you are running containers in Kubernetes there is the option to set the time zone of the Kubernetes pod which can act as a temporary workaround: https://mohammedomar.net/2020/08/18/changing-timezone-for-kubernetes-windows-based-nodes/ https://github.com/MikkelHegn/k8-deployments/tree/main/other_demos/set_timezone |
This issue has been open for 30 days with no updates. |
1 similar comment
This issue has been open for 30 days with no updates. |
May we get an update on when this will be available? We are desperately looking for ability to change windows container timezone to what's desired. Regarding the workaround provided by you, we tried this in a dev environment and it seems to be working fine. Can you confirm if this is safe to be done in a production environment too? |
Hi everyone, apologies for the delayed update! Our team was unfortunately hit with some high sev issues that entirely consumed our development cycles and delayed the completion of this feature. Nonetheless the feature is complete and we are working through validation and getting the feature through OS servicing. Right now we are on track to hit the 4C servicing deadline, which translates to a release in late April. I understand this is later than expected, but again we are working to get this feature out ASAP. I will be sure to update this thread again once the feature is on its way through servicing. Feel free to reach out to me if you have concerns - I am more than happy to work something out if needed. For more info on servicing: @SachinL9 the workaround there simply creates a K8s DaemonSet that sets your preferred time zone of each node in your cluster. The current behavior of Windows server containers is that the container shares the time zone settings of the host, so if you set the time zone of a host it will be reflected by the container. So with the DaemonSet workaround every pod deployed should share the same configured time zone. With the change we are introducing in this feature the time zone will be completely virtualized within the container. When you launch a container it will mirror the TZ settings of the host, but you can configure the container's time zone from within the container using tzutil or Set-Timezone in powershell. Doing this requires a significant number of kernel changes, which is why it's been taking time to get this feature out. Again if you have other questions, don't hesitate to reach out to me! |
Hi, I read that virtualized timezones will be available in the upcoming Windows Server 2022, but I am assuming the feature being worked on in this issue will not require Windows Server 2022, is that correct? Sounds like the work done here will be brought into 2022 :) Thanks again! |
Hey everybody, sending another quick update that we are still on track to hit the 4C release! The feature required a significant number of modifications to both the Windows kernel and guest OS so we've been running it through rigorous testing to ensure the feature works properly. It should also be noted that the serviced container images are not produced externally until the following B release, which in this case will be about mid-May. This feature requires both an updated host and guest to function properly. I will be putting together detailed documentation on this in the next few weeks. @yylai that is correct! We developed this feature as part of the mainline OS and then backported it to WS2019 and other intermediary releases. If you'd like to test it out the feature is available to insiders today. And again, more documentation will follow over the next few weeks. |
Hey @brasmith-ms, thats great to hear! Not sure if I'm the only one who gets confused about this but could you (or anyone else) clarify will this feature be released into the LTSC channel or SAC channel? If LTSC will MSFT be updating the tag from Also am I correct in thinking that we cannot run Insider releases on AKS? Windows container support requires the host OS to be updated to the same level IE our nodepools would need to also be on the insider track which we cannot enable in AKS currently. |
@GerryWilko good points! This feature will be available on 1809 (server 2019 LTSC), 2004, and 20h1 (for reference). It will not be available on 1903 or 1909 since both will be EOL by the time this is released. It will also be available on Windows server 2022 once it GAs. Basically, all mainstream supported versions will have this feature 🙂 And you are correct, at the moment insider releases cannot be run on AKS so use of this version would purely be for dev / testing purposes. If there are any specific concerns on this please reach out to me, happy to work something out. |
Hi folks! Jumping in here with another update. The feature is complete and was checked in and tested for the 4C release. An important thing to note, however, is that this feature requires both the host and guest to be updated to function properly. Per the Windows servicing process container images are only released during B releases, so the feature was released for 4C but guest images won't be available publicly until 5B (May patch Tuesday, the 11th). I appreciate your patience in waiting for this and am anxious to get this feature into your hands. We will be releasing documentation for the feature over the next week or so as well, so stay tuned! |
That great @brasmith-ms. Congrats to the team! What does this mean for AKS? Do we need to wait for an Kubernetes version bump? I don't think we have options to select node pool OS version. Cheers! |
Might have found the answer here actually. Azure/AKS#2295 Seems to be rolling out already. |
This seems to be working now. From a colleague's testing, no luck with tzutil but Set-TimeZone works. Thanks! |
@sicklittlemonkey what node image are you using? I just upgraded to the latest version and still got a Access Denied error. |
@remenaker - Our node OS image reference from the portal is:
We use the following base image and tag in our Dockerfile: |
@sicklittlemonkey - Just to make sure I'm barking up the right tree, can you confirm that you've seen this work in AKS or Kubernetes? Just trying to confirm all the pieces that I need to worry about between the application's base image and the node's OS image. Any details would be great. |
@cwiederspan - We use Docker with Service Fabric, but yes, Set-TimeZone seems to be working (but not tzutil).
Note that tzutil /s appears to change the time zone without error, but the time doesn't change:
|
|
FWIW, I still cannot get this to work in Azure Kubernetes (AKS). My application image using the latest base images now builds fine, it works locally, but in AKS the time zone in the container is not set/persisted. I suspect it has to do with the node image being out of date. I think @brasmith-ms is also working this angle, so maybe he'll provide an update if/when he's got something. |
Actually, I was mistaken. The Windows message is sent either way. The issue @sicklittlemonkey reported is that PowerShell relies on the cached data in You can see this yourself if after the example @sicklittlemonkey showed, call That happens regardless of whether running in a container or not. My recommendation is that if you are using PowerShell and continuing with other commands in the same session, that you use |
Hi everyone! Yes the V-TZ feature is now available on MCR. For Windows server 2019 the image version is 10.0.17763.1935, and for the latest SAC release it is 10.0.19042.985. Note that both the host and container image must be at least this version to work properly as the feature had both kernel and user space modifications. I'll be publishing more detailed documentation on MSDN in the next week or so to help detail the differences. The gist of this is that the best way to set the time zone in the container is through tzutil or powershell Set-TimeZone. For images without either of these functions and utility that calls the SetDynamicTimeZoneInformation Win32 API will be necessary. For the folks running on AKS I apologize for the confusion here. AKS takes some time to ingest new servicing releases and they are currently working on building a new AKS Windows VHD based on the 5B servicing release which should be included in next week's AKS RP release (5/20 target). Once this is available the V-TZ feature will be fully functional using the latest MCR images. |
It seems like that both |
Hi, @brasmith-ms, Do you have an update for us on this feature? I am still unable to get this feature to work on AKS and without the docs for this feature its unclear what we need to do to get it to work. Currently running the RUN Set-Timezone -Name 'GMT Standard Time' At the moment as far as I can tell I have patched our build agents to latest available version (currently 10.0.17763 Windows Server 2019 Datacenter). AKS is showing AKSWindows-2019-17763.1879.210414 with latest release even after a full node recycle. One thing that would be great to clear up is does this Timezone patch need to be run each time a container starts? Does the built image retain this virtualised setting or does this need to be executed in an |
@GerryWilko, which region are you using? The latest AKS Windows VHD should be 17763.1935.210513 and Set-TimeZone only works in this VHD with 5B. |
Hi @AbelHu, I am in North Europe for my AKS. Cheers |
|
Yeah I just found this help article. https://docs.microsoft.com/en-us/azure/aks/node-image-upgrade Running the upgrade now to see if this resolves the issue. Many thanks! |
@GerryWilko - With a little help from @brasmith-ms, I was able to confirm on Friday that this was all working for me on AKS. My approach was this...
As I said, this all worked for me. In reality, I wouldn't want to hard-code the time zone into the Dockerfile, and would instead want to set it at container creating time based on an env or config variable, otherwise I'd need a different image for every time zone (duh!). |
Hi @cwiederspan, Thanks! Strange then, I'm starting to think that I have not patched the build agents to the right level to make this work. Can you confirm what OS version you built your Dockerfile on? Powershell |
I built the container image on Azure Container Registry using an The one thing that I did struggle with was ensuring that the AKS node's OS image was up to date with the latest version. To do so, I ran the aforementioned
That then shows me some output like this... ...
"mode": "User",
"name": "win001",
"nodeImageVersion": "AKSWindows-2019-17763.1935.210513",
"nodeLabels": null,
"nodePublicIpPrefixId": null,
"nodeTaints": null,
"orchestratorVersion": "1.20.5",
... Once I saw that nodeImageVersion showing the new .1935. version, then things started to work. |
Thanks for this @cwiederspan! Ok so it looks like it should all be working but we just need to know when to expect this 5B servicing release in normal windows updates channels so that we can patch our on-premise build agents. @brasmith-ms @AbelHu would that be right? Is there any way I can similarly force upgrade of my Windows Server 2019 LTSC build agents to pull the 5B patch? |
@GerryWilko the 5B update is already available and live via the normal Windows Server Update Service / Windows update channels so you should just be able to follow the regular process. Let me know if this doesn't work for you. https://support.microsoft.com/en-us/topic/may-11-2021-kb5003171-os-build-17763-1935-3f03e74b-4759-4ca3-b9f1-4bc0d5ab5d27 |
Hey everyone! We have published the documentation for this feature. Thanks again to everyone for your patience, please let me know if you hit any problems or have additional suggestions! |
Currently Windows containers use the host time zone by default. If a user wants to change the time zone they must change the time zone of the host. Running the tzutil executable in a Windows Server container will simply throw a lack of privileges error, and registry time zone settings are ignored upon starting the container. The only way of ensuring that the time zone is set properly is through configuration of the host – which defeats the package-and-ship paradigm of containers.
This issue has been discussed in detail here.
Upon the release of the fix for this issue users will be able to use the tzutil command, Set-TimeZone powershell cmdlet, or set the virtual registry key.
The text was updated successfully, but these errors were encountered: