New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
v0.20.0 -once doesn't wait for templates to finish #1196
Comments
Tested this against master ( ebab455 ), not working there either. |
|
@pearkes any ideas what could be causing this? |
Did some testing and turns out (The branch is not really PR quality, but provides a proof-of-concept for fixing the bug.) EDIT: |
Another view of the issue is this: errnoh@633c4f9 (this change fixes my issue and enables (But this probably causes issues somewhere else in the code) |
@errnoh interesting, yeah when I tried similar approaches was met with breaking other things that assumed this loop. The test suite is pretty solid, so running that on your branch should validate beyond the use-case quite well. |
Seeing the same behavior with --once. Looks like process completes properly without actually writing output. Same config produces output without --once flag. Interestingly -once will not complain about write permissions on output either. In this case /etc/haproxy/haproxy.cfg should not be writeable and consul-template recognizes this without -once flag. 2019/04/15 15:45:38.195442 [INFO] (runner) creating watcher
2019/04/15 15:45:38.210924 [INFO] (runner) starting
2019/04/15 15:45:38.210936 [DEBUG] (runner) running initial templates
2019/04/15 15:45:38.210941 [DEBUG] (runner) initiating run
2019/04/15 15:45:38.210976 [DEBUG] (runner) checking template 16d5900abcded9dbd7
2019/04/15 15:45:38.211208 [DEBUG] (runner) was not watching 1 dependencies
2019/04/15 15:45:38.211226 [DEBUG] (watcher) adding kv.block(web/webapps/busybox)
2019/04/15 15:45:38.211232 [TRACE] (watcher) kv.block(web/webapps/busybox) starting
2019/04/15 15:45:38.211238 [DEBUG] (runner) checking template cf132b31147013b714
2019/04/15 15:45:38.211325 [DEBUG] (runner) was not watching 1 dependencies
2019/04/15 15:45:38.211331 [DEBUG] (watcher) adding kv.block(web/test)
2019/04/15 15:45:38.211336 [TRACE] (watcher) kv.block(web/test) starting
2019/04/15 15:45:38.211341 [DEBUG] (runner) diffing and updating dependencies
2019/04/15 15:45:38.211351 [DEBUG] (runner) enabling global quiescence for "10590030d69abcded9dbd7"
2019/04/15 15:45:38.211357 [DEBUG] (runner) enabling global quiescence for "cf1320c71147013b714"
2019/04/15 15:45:38.211361 [DEBUG] (runner) watching 2 dependencies
2019/04/15 15:45:38.211404 [TRACE] (view) kv.block(web/test) starting fetch
2019/04/15 15:45:38.211424 [TRACE] kv.block(web/test): GET /v1/kv/web/test?stale=true&wait=1m0s
2019/04/15 15:45:38.211897 [TRACE] (view) kv.block(web/webapps/busybox) starting fetch
2019/04/15 15:45:38.211912 [TRACE] kv.block(web/webapps/busybox): GET /v1/kv/web/webapps/busybox?stale=true&wait=1m0s
2019/04/15 15:45:38.228984 [TRACE] kv.block(web/webapps/busybox): returned "10.x.x.x:80"
2019/04/15 15:45:38.228998 [TRACE] (view) kv.block(web/webapps/busybox) marking successful data response
2019/04/15 15:45:38.229006 [TRACE] (view) kv.block(web/webapps/busybox) successful contact, resetting retries
2019/04/15 15:45:38.229011 [TRACE] (view) kv.block(web/webapps/busybox) received data
2019/04/15 15:45:38.229018 [DEBUG] (runner) receiving dependency kv.block(web/webapps/busybox)
2019/04/15 15:45:38.229028 [DEBUG] (runner) initiating run
2019/04/15 15:45:38.229032 [DEBUG] (runner) checking template 16d0c605ded9dbd7
2019/04/15 15:45:38.229143 [DEBUG] (runner) checking template cf1320c713b714
2019/04/15 15:45:38.229219 [DEBUG] (runner) missing data for 1 dependencies
2019/04/15 15:45:38.229225 [DEBUG] (runner) diffing and updating dependencies
2019/04/15 15:45:38.229229 [DEBUG] (runner) kv.block(web/webapps/busybox) is still needed
2019/04/15 15:45:38.229233 [DEBUG] (runner) kv.block(web/test) is still needed
2019/04/15 15:45:38.229238 [DEBUG] (runner) watching 2 dependencies
2019/04/15 15:45:38.229295 [TRACE] kv.block(web/test): returned "foozballzzzz"
2019/04/15 15:45:38.229301 [TRACE] (view) kv.block(web/test) marking successful data response
2019/04/15 15:45:38.229307 [TRACE] (view) kv.block(web/test) successful contact, resetting retries
2019/04/15 15:45:38.229311 [TRACE] (view) kv.block(web/test) received data
2019/04/15 15:45:38.229317 [DEBUG] (runner) receiving dependency kv.block(web/test)
2019/04/15 15:45:38.229321 [DEBUG] (runner) initiating run
2019/04/15 15:45:38.229324 [DEBUG] (runner) checking template 16d0c6059ed9dbd7
2019/04/15 15:45:38.229415 [DEBUG] (runner) checking template cf1320c785d73b714
2019/04/15 15:45:38.229488 [DEBUG] (runner) diffing and updating dependencies
2019/04/15 15:45:38.229494 [DEBUG] (runner) kv.block(web/webapps/busybox) is still needed
2019/04/15 15:45:38.229498 [DEBUG] (runner) kv.block(web/test) is still needed
2019/04/15 15:45:38.229503 [DEBUG] (runner) watching 2 dependencies
2019/04/15 15:45:38.229506 [DEBUG] (runner) all templates rendered
2019/04/15 15:45:38.229510 [INFO] (runner) once mode and all templates rendered
2019/04/15 15:45:38.229513 [INFO] (runner) stopping
2019/04/15 15:45:38.229517 [DEBUG] (runner) stopping watcher
2019/04/15 15:45:38.229520 [DEBUG] (watcher) stopping all views
2019/04/15 15:45:38.229524 [TRACE] (watcher) stopping kv.block(web/webapps/busybox)
2019/04/15 15:45:38.229528 [TRACE] (watcher) stopping kv.block(web/test)
note: hashes above have been munged. to be clear, works fine in daemon mode. |
I am also getting this same issue. I have a few consul-template config files that are meant to fetch certificates from vault/pki and write them to disk.
Please let me know if there's anything else i can do to help. |
Hey @errnoh and everyone. While playing with this I got to thinking about when you would want to use wait and once together and haven't really been able to come up with a good reason. Assuming wait was added to prevent template re-rendering and possible subsequent reloading of the process when a watched value was flapping... how does this make sense when using once? Basically I'm trying to come up with a reason why the fix for this couldn't just be to make -once disable waits/quiescence. Thanks. |
As for my comment on the related cases, I think there’s a complex of problems here ever since the change queue code was refactored around the 0.19.2 release. Personally I agree that -once could disable -wait; this would put the onus of splay times back onto a caller script for example. But I don’t think it solves any of the related problems where the quiesce times aren’t being respected or rendering gets stuck, etc. |
I 100% agree that there are more problems here than just this once/wait issues. I'm not looking to address it here for a couple reasons. First is that I don't think I've had enough time working with the code to really feel confident about tackling that just yet. I'm working up to it. Second, I do think that having once disable wait/quiescence is the best way to address this bug as it eliminates a code path (simplifies things) and is closer to what I'd think the expected behavior would be. Thanks. The feedback is much appreciated. |
Are there any particular issues, aside from these wait/once ones (#1196, #1207, #1209), that you would call out as fallout from this refactor? Are there behaviors or problems that you'd call out that don't have issues filed for them yet? Thanks. |
Consul Template version
consul-template v0.20.0 (b709612)
Configuration
Command
consul-template -once -log-level=trace -config="/tmp/consul-template.hcl"
Debug output
Running without
-once
pauses for five seconds after theall templates rendered
line and then continues with the following:This results with the template properly rendered to target location.
Expected behavior
/tmp/foobar.pem
is written to disk. This works fine in v0.19.5.Actual behavior
consul-template -once exits before the template is rendered.
Steps to reproduce
consul-template -once
with version v0.20.0References
The text was updated successfully, but these errors were encountered: