Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

timeX.google.com provide non standard time #437

Closed
isomer opened this issue Jun 30, 2015 · 62 comments
Closed

timeX.google.com provide non standard time #437

isomer opened this issue Jun 30, 2015 · 62 comments
Labels

Comments

@isomer
Copy link

@isomer isomer commented Jun 30, 2015

systemd should not default to using time{1,2,3,4}.google.com.

Google doesn't provide timeX.google.com as a public service(0). We don't maintain this to the same level of reliability/availability that we work hard to provide with other services (eg Google Public DNS, websearch etc). We use this for systems outside of our datacenters that need to understand our concept of time.

Google (famously(1)) runs a non standard concept of time, that works well if everything in your infrastructure knows and understands this and is designed to work with this. If you mix and match NTP servers within a machine you'll run into problems (since Google's timeservers are deliberately false ticking). If you mix and match between machines (eg some hosts use Google's time servers, and some use NIST timeservers) then your time won't match up between them. As I write this Google's clock is ≈-0.4s out from UTC.

Services such as NFS are likely to be confused as to the creation times of files and so on, logs won't be comparable between hosts etc.

People shouldn't be using these time servers for their own use. People should definitely /not/ be using these time servers by default without realising it.

Please change systemd's default time servers to some service that is designed to provide a reliable public NTP service.

(0): http://googlecloudplatform.blogspot.com/2015/05/Got-a-second-A-leap-second-that-is-Be-ready-for-June-30th.html
(1): http://googleblog.blogspot.com/2011/09/time-technology-and-leaping-seconds.html

@gobengo
Copy link

@gobengo gobengo commented Jul 1, 2015

I suppose the obvious question is, what's a better option? Or what are even choices for another option? Does this apply? http://www.pool.ntp.org/en/

@codercotton
Copy link

@codercotton codercotton commented Jul 1, 2015

pool.ntp.org?

@blt
Copy link

@blt blt commented Jul 1, 2015

@gobengo Yes, the NIST timeservers isomer is referring to in the original post are available publicly through the ntp.org pool. These are the public NTP servers most machines peg themselves to, outside of organizations that run their own pools etc etc.

@hikari-no-yume
Copy link

@hikari-no-yume hikari-no-yume commented Jul 1, 2015

time.windows.com?

@jleclanche
Copy link

@jleclanche jleclanche commented Jul 1, 2015

From http://support.ntp.org/bin/view/Support/WindowsTimeService:

Stand-alone Windows servers and clients are automatically configured to poll time.windows.com at one-hour intervals. The time.windows.com server (actually a cluster of servers) is maintained by Microsoft. However, time.windows.com is notoriously unreliable and heavily loaded, so configuring a different time source or multiple sources is probably wise.

Maybe not a wise idea :) What's wrong with pool.ntp.org?

@jedsmith
Copy link

@jedsmith jedsmith commented Jul 1, 2015

timesyncd just needs this, that's all:

This is a problem beyond systemd, too. Router manufacturers are notorious for throwing burned-in traffic at NTP systems that do not expect it; this issue is just another variant of that longstanding one.

Since this is a publicly-linked issue with some traffic now, I'll soapbox for a moment: If you run NTP for your own organization, do it correctly, please. Either use the NTP pool up to a certain size (I'd say a couple dozen systems), or configure your own edge stratum 3 servers that sync from publicly-available stratum 2, then sync your own network off those. And put your S3s in the pool.

If you are pushing configuration management to hardcode time.nist.gov, *.bldrdoc.gov, time.windows.com and friends across your hundreds of machines, you are absolutely doing it wrong.

@floppym
Copy link
Contributor

@floppym floppym commented Jul 1, 2015

For some reason Lennart was opposed to requesting a vendor zone last year.

http://lists.freedesktop.org/archives/systemd-devel/2014-August/022575.html

His reasoning seems rather stupid to me. In any case,

using Google's servers without their backing/consent seems bad.

@jedsmith
Copy link

@jedsmith jedsmith commented Jul 1, 2015

@floppym The PR stops short of a vendor zone for that reason, and I agree.

@crrodriguez
Copy link
Contributor

@crrodriguez crrodriguez commented Jul 1, 2015

You cannot use pool.ntp.org either.. "You must absolutely not use the default pool.ntp.org zone names as the default configuration in your application or appliance." that's according to the pool's website...

@abh
Copy link

@abh abh commented Jul 1, 2015

@blt I don't believe the NIST servers are part of the NTP Pool.

@floppym Lennarts reasons for not using *.systemd.pool.ntp.org ("systemd isn't a distribution") makes sense to me.

@jleclanche As far as I know Microsoft's NTP service at time.windows.com is perfectly respectable these days. Years ago it was run by a third party vendor and it was pretty terrible.

@crrodriguez If you kept reading you'd learn about getting a "vendor pool" setup specifically for your product/distribution.

I'd suggest having no default NTP servers in the systemd code; leave it to distributors to provide their own (possibly via the NTP Pool).

@blt
Copy link

@blt blt commented Jul 1, 2015

@abh, would you look at that. Right you are. That's a time infrastructure misunderstanding corrected for me today. Thanks!

On Jun 30, 2015, at 8:14 PM, Ask Bjørn Hansen notifications@github.com wrote:

@blt I don't believe the NIST servers are part of the NTP Pool.

@floppym Lennarts reasons for not using *.systemd.pool.ntp.org ("systemd isn't a distribution") makes sense to me.

@jleclanche As far as I know Microsoft's NTP service at time.windows.com is perfectly respectable these days. Years ago it was run by a third party vendor and it was pretty terrible.

@crrodriguez If you kept reading you'd learn about getting a "vendor pool" setup specifically for your product/distribution.

I'd suggest having no default NTP servers in the systemd code; leave it to distributors to provide their own (possibly via the NTP Pool).


Reply to this email directly or view it on GitHub.

@xorgy
Copy link

@xorgy xorgy commented Jul 1, 2015

Has anyone applied for a vendor zone? I think systemd, although not a "distribution" could be considered a vendor or product.

Even if technically distributions need to think about this, if you have a default configuration(and you should) then it should be not unreasonable to leave it untouched, including not changing the NTP servers.

I don't see any reason not to apply for a vendor zone either way.

@floppym
Copy link
Contributor

@floppym floppym commented Jul 1, 2015

Lennarts reasons for not using *.systemd.pool.ntp.org ("systemd isn't a distribution") makes sense to me.

This isn't about "providing a distribution". It's about configuring a useful default for people who build/configure systemd from the git repo or source tarball. The people who run the NTP pool just want that on file.

Anyway, I agree that having no default is better than picking some random NTP servers like Google or using the public NTP pool without a vendor zone.

@jedsmith
Copy link

@jedsmith jedsmith commented Jul 1, 2015

I think "no default" versus "stop the bleeding and get off Google per Google's request" is a theoretical discussion instead of a practical one, given that systemd has already shipped a default. Every build will break if that behavior changes. I agree that no default would be better, but I suspect that ship has sailed.

I'll leave my PR up in anticipation of being correct. I hope @abh forgives me for breaking the very FAQ I pointed folks toward in my PR given those circumstances.

@StoneCypher
Copy link

@StoneCypher StoneCypher commented Jul 1, 2015

if you don't take this down, google will just remove the dns, and then you lose the ability to do this in a controlled way

it does not matter if the ship has sailed; call it back. we haven't left someone behind on fare. the ship is dangerous and unreliable.

@floppym
Copy link
Contributor

@floppym floppym commented Jul 1, 2015

Every build will break if that behavior changes.

Not so. I would expect that most Linux distros already override it via the configure switch.

@jedsmith
Copy link

@jedsmith jedsmith commented Jul 1, 2015

@floppym You hedged that statement yourself, making the assertion a nonstarter as a software project.

@floppym
Copy link
Contributor

@floppym floppym commented Jul 1, 2015

Saying you can never change something because you did something stupid in the past with makes no sense at all.

You either make timesyncd work out of the box properly, or you force people to configure it if they happen to build systemd themselves.

@angelbladex
Copy link

@angelbladex angelbladex commented Jul 1, 2015

i use 0.arch.pool.ntp.org on Arch but dont use systemd with ntp: a simple script on crontab

@gryftir
Copy link

@gryftir gryftir commented Jul 1, 2015

http://www.pool.ntp.org/en/use.html lists 4 servers which are randomized and changed every hour. Those should be the defaults.

@xorgy
Copy link

@xorgy xorgy commented Jul 1, 2015

Arch indeed has its own timeservers {0..3}.arch.pool.ntp.org which it configures into the distribution systemd package, as does Fedora {0..3}.fedora.pool.ntp.org, as does debian {0..3}.debian.pool.ntp.org, as does ubuntu {0..3}.ubuntu.pool.ntp.org, and gentoo, and opensuse...

I think distributions are pretty good about specifying these, so maybe the defaults don't really need to be sane, just sane enough that they work.

I think maybe {0..3}.pool.ntp.org is okay for developer builds without configuration.

@poettering
Copy link
Member

@poettering poettering commented Jul 1, 2015

Its up to the distros to register an ntp pool product. systemd is not a product. Its just some toolset people can build products from. We cannot use the ntp pool hence. If the googl time servers

@poettering poettering closed this Jul 1, 2015
@floppym
Copy link
Contributor

@floppym floppym commented Jul 1, 2015

@poettering If systemd is not a "product" that is meant to work on its own, then there should be no need to configure any default NTP servers since nobody should ever utilize it that way.

Using Google's servers by default is still a valid problem and this issue should be reopened.

@StoneCypher
Copy link

@StoneCypher StoneCypher commented Jul 1, 2015

@poettering it is not appropriate for you to close the issue by google saying "stop using our resources" because you have decided it is inconvenient.

you do not have the privilege of using their servers over their protest. choices of this form do serious damage to community good will. please re-open.

@GaiusCoffee
Copy link

@GaiusCoffee GaiusCoffee commented Jul 1, 2015

Both timeX.google.com and systemd are both non-products, therefore, there's no problem here. Right?

@johnku1
Copy link

@johnku1 johnku1 commented Jul 1, 2015

Curious, why is a FOSS project depending on a proprietary time server(i.e Google's) as a first resort?

@mlindner
Copy link

@mlindner mlindner commented Jul 1, 2015

@poettering You're being kind of absurd and stupid here. Supplying a broken timeserver as a default is inane and silly. Please accept the pull request. #439

@jhalterman
Copy link

@jhalterman jhalterman commented Jul 1, 2015

Looking forward to seeing the emergency update after Google pulls the DNS entry for its time servers.

@hishamhm
Copy link

@hishamhm hishamhm commented Jul 1, 2015

@poettering Your explanation makes sense, except for the fact that you are ignoring Google's request for having their server being removed from the source code as a default. It's not a matter if their servers are bad or not, it's a gesture of goodwill towards the request from someone providing a publicly accessible resource.

@dannyperson's link above pretty much settles that ntp.org is fine with having pool.ntp.org as a default in the source code as long as vendors change it when shipping (as they do).

@jedsmith
Copy link

@jedsmith jedsmith commented Jul 1, 2015

@abh Can you perhaps speak to this perceived corner case in the usage of the pool? This seems to be a situation not encountered before, since traditional ntpd does not ship with defaults and instead only has example configurations with ACTS drivers and such.

We're obviously at an impasse here. Allow me to summarize:

  • Google firmly requests, including with emphasis, that systemd stop using time?.google.com.
  • @poettering is firmly against registering as a vendor, and you agree. CoreOS has expressed willingness to take on the near-zero burden of doing so (and we've filed ntppool #4457).
  • @poettering interprets the remainder of your documentation as hard forbidding systemd's usage without registering as a vendor. I haven't seen your opinion on this.

There are only three paths forward, given all of the cards on the table:

  • Remove defaults and break testing/one-off builds of systemd.
  • Hardcode another server instead of Google's, and hope it never breaks. The public StratumTwo list from ntp.org will be useful for this, but it's yet another server administrator to bother.
  • Leave Google in as the default explicitly against their request (I consider this a nonstarter).

Can you share your perspective on whether the NTP pool can help at all here as a fourth option?

@martinpitt
Copy link
Contributor

@martinpitt martinpitt commented Jul 1, 2015

After reading that, it seems to me that this would be a viable compromise:

  • Don't default to any server.
  • Skip timesyncd tests with a big warning if you configured without any
  • In semaphore and similar CI tests, configure with an appropriate server. E. g. semaphore runs under Ubuntu, and it would be perfectly valid to use ?.ubuntu.pool.ntp.org, and I don't think anyone would complain if it used the Debian/Arch/whatever pool.

This avoids having to take responsibility for a "systemd" pool (I agree that this makes no sense at all and we don't want that), respects Google's wishes and we don't have so many systemd CI systems out there that it's a big burden to change their config options (essentially just semaphore, I think -- distributions will already use their own configure options for their CI).

@martinpitt martinpitt reopened this Jul 1, 2015
aneeshusa added a commit to aneeshusa/systemd that referenced this issue Jul 1, 2015
Google's NTP servers do not provide UTC time but 'Google Time' so it is
not appropriate to use them as default. As we are not a vendor, do not
use a vendor pool. Instead, provide no default and require distributions
to provide ntp server values.
@aneeshusa
Copy link

@aneeshusa aneeshusa commented Jul 1, 2015

This PR takes away the default ntp servers (I agree a systemd vendor pool doesn't make sense.)

EDIT: Oh, I didn't see your comment about skipping timesyncd tests, I should add that to the PR.

@aneeshusa
Copy link

@aneeshusa aneeshusa commented Jul 1, 2015

Actually, what's the use case for configuring with timesyncd but not providing ntp servers (i.e. when would we skip timesyncd tests)? CI can just pass the option, and I can't think of another use case at 2:30 AM...

@martinpitt
Copy link
Contributor

@martinpitt martinpitt commented Jul 1, 2015

Sure, I'm also fine with tests just failing if you forgot to configure with an NTP server. (FTR, I'm not even aware of any timesyncd tests upstream..).

@aneeshusa
Copy link

@aneeshusa aneeshusa commented Jul 1, 2015

I can't find any timesyncd tests, so I'm not going to add code to skip something that doesn't exist.

@kekimmo
Copy link

@kekimmo kekimmo commented Jul 1, 2015

@dannyperson @hishamhm I don't understand that interpretation.

Open Source projects are of course particularly welcome to use the pool in their default setup, but we ask that you get a vendor zone when using the pool as a default configuration.

Isn't that specifically asking Open Source projects ("you") to get a vendor zone if they use the NTP pool in their default configuration?

So, if systemd wants to use the NTP pool in systemd's default configuration, systemd is particularly welcome to do so, but systemd must get a vendor zone.

It seems to me that "vendor zone" is just a name for a concept, and it's simply an accident of history that it isn't called "default config zone". Correct me if I'm wrong, but isn't arguing that systemd is not a vendor like arguing that you can't wear aviator sunglasses if you're not a pilot?

@cathalgarvey
Copy link

@cathalgarvey cathalgarvey commented Jul 1, 2015

@kekimmo Worse, the idea is to avoid setting up a sane default because insane defaults (Google) exist and were/are preconfigured.

The two solutions here are obvious:

  1. Set up a sane preconfigured option if a preconfigured option is to exist at all: Replace Google with systemd.pool.ntp.org, allowing CoreOS to register it so that it feels comfortably NIH to people who object to self-vendoring (???).
  2. Remove defaults and make the build halt and catch fire if a sane NTP server isn't provided, so downstream users know they must provide for themselves. Have a flag for testing.

The first seems to require the least work; just change the default to a sane one and move on.

@MatejLach
Copy link

@MatejLach MatejLach commented Jul 1, 2015

If @poettering doesn't want to vendor a pool, doesn't want the NTP servers to be blank by default and Google doesn't want systemd to use their test servers, the only possible solution I could see is for ntp.org to provide some sort of devtest.pool.ntp.org if they are willing to, so that systemd and other projects that feel similarly can use it as a reasonable default without annoying anyone.

I don't think that CoreOS registering systemd.pool.ntp.org is the best move. as to me it sounds like @poettering simply doesn't want any end-users to view systemd as some sort of a product providing its own services and, (at least in the minds of some users), being assigned the responsibility for those services or start treating systemd as a separate product, instead of it being an integral part of the distribution you're using.

I, personally, can sort-of see the reasoning behind this; An analogy: If you're ie a supplier of a physics library for game developers to use, you don't want gamers, (end-users), to start treating your library like a separate product that they can call, rely on etc. - this is the sort of realationship systemd seems to see itself in with Linux distributions; we are not the end product and therefore are not providing any services for end-users to rely on.

So, to make everyone happy, I think the solution is fairly simple:

  1. A third-party needs to provide a "neutral" NTP server that systemd can use by default and not have it be associated with systemd as in "being provided by/a part of" systemd, (I assume that they don't want to use RedHat's servers here to not promote the idea that systemd is being "controlled" by RedHat).
  2. systemd needs to adopt the neutral NTP servers ASAP.
  3. Everyone needs to calm down.
@mburnicki
Copy link

@mburnicki mburnicki commented Jul 1, 2015

Here is a plot how time1.google.com smeared the time over the leap second:
http://people.ntp.org/burnicki/leap-second-2015-06-30/time1.google.com-216.239.32.15.png

And here is what time.windows.com was doing:
http://people.ntp.org/burnicki/leap-second-2015-06-30/time.windows.com-23.99.222.162.png

So I's say neither of them is appropriate for accurate timekeeping.

@systemd systemd locked and limited conversation to collaborators Jul 1, 2015
@poettering
Copy link
Member

@poettering poettering commented Jul 1, 2015

I have now deleted a couple of hate-filled messages here and locked the issue. I'll unlock it again in a few days when the reddit peanut gallery lost interest.

@zonque zonque added the timesync label Jul 1, 2015
@systemd systemd unlocked this conversation Jul 9, 2015
@systemd systemd unlocked this conversation Jul 9, 2015
poettering added a commit to poettering/systemd that referenced this issue Jul 11, 2015
…uilding systemd

Also, explain the situation in the docs.

Relates to systemd#437
@poettering
Copy link
Member

@poettering poettering commented Jul 11, 2015

I have now unlocked this issue again. I have also put together PR #554, which adds a configure time warning, and documentation for the issue. If people leave the default google ntp servers in, they'll now get a configure time warning.

@isomer does this make sense to you? Again, I am pretty sure we should not register systemd as a vendor pool project with ntp.org since downstreams really should register their own individual ones, and nobody should use a systemd pool at all. OTOH we also would like to ship something that allows people to test systemd without having to register/pick an ntp.org vendor pool. Hence I chose the Google NTP servers. For this usecase it doesn't matter really if they implement a non-standard clock, and it doesn't matter either if it's a service that can go away any day.

Of course, if Google explicitly doesn't want us to use the service even for this kind of testing setups, even if we acknowledge that it isn't accurate and not officially supported, we could remove the reference entirely. Can you elaborate on this?

Again, all downstream distros I am aware of have their own ntp.org vendor pools anyway, and set them at ./configure time. Only folks who build everything from source on their own, without using any kind of prepackaged distro would run with these defaults. Hence the default should really doesn't matter too much in real life.

@jleclanche
Copy link

@jleclanche jleclanche commented Jul 12, 2015

@poettering Given that the server would near-exclusively be used for testing, is there an actual reason the default (non-vendor) ntp.org servers won't do? FWIU, they are completely fine for personal and testing use. It just seems like there would be very low usage of them, especially now that a warning is being added.

@MatejLach
Copy link

@MatejLach MatejLach commented Jul 12, 2015

@jleclanche I believe that systemd using non-vendored ntp.org servers would unfortunately be explicitly against their policy.

You must get approval from the server operator before you hardcode any IP addresses or hostnames. 
This is easy to get if your own organization runs the NTP servers you are planning to use. In most other cases you will not get it.

Do not use the standard pool.ntp.org names as a default configuration in your system. The NTP Pool can offer services for you, but it must be setup in advance (see below).

Source: http://www.pool.ntp.org/en/vendors.html

@StoneCypher
Copy link

@StoneCypher StoneCypher commented Jul 14, 2015

@poettering Thank you for allowing this discussion to continue

@poettering
Copy link
Member

@poettering poettering commented Jul 23, 2015

Hmm, I fear we scared @isomer away. I will close this issue now, given that #554 has been merged now, which settles the issue from my perspective. @isomer, I'd still be interested in your feedback, and we can reopen the issue, should there be something left to fix.

@poettering poettering closed this Jul 23, 2015
@alfiepates
Copy link

@alfiepates alfiepates commented Jul 24, 2015

@poettering

I'm not entirely satisfied with this solution. Warnings get ignored, and you're still using timeX.google.com as default.

If you're hardcoding NTP servers, it's only fair that you take responsibility for those servers.

@cathalgarvey
Copy link

@cathalgarvey cathalgarvey commented Jul 24, 2015

I may be merely part of the "peanut gallery" of /r/linix but FWIW I also think doing nothing about this is foolish. Let another project generously allocate a pool as offered, and use it. One way or another, you're using someone else's NTP, but the current way is unfair, brittle, and likely unsustainable.

On 24 July 2015 11:48:28 GMT+01:00, Alfie Pates notifications@github.com wrote:

@poettering

I'm not entirely satisfied with this solution. Warnings get ignored,
and you're still using timeX.google.com as default.

If you're hardcoding NTP servers, it's only fair that you take
responsibility for those servers.


Reply to this email directly or view it on GitHub:
#437 (comment)

Sent from my Android device with K-9 Mail. Please excuse my brevity.

@lewiseason
Copy link

@lewiseason lewiseason commented Jul 24, 2015

Likewise. Alternatively, ask pool.ntp.org for clarification as to whether this is a legitimate use case for the default pool. The wording isn't terribly clear - and it would be nice to clear up any ambiguity - especially if you're dead set against using a vendor pool

@systemd systemd locked and limited conversation to collaborators Jul 24, 2015
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Linked pull requests

Successfully merging a pull request may close this issue.

You can’t perform that action at this time.