Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.service subpkg #483

Merged
merged 38 commits into from
Mar 10, 2023
Merged

.service subpkg #483

merged 38 commits into from
Mar 10, 2023

Commits on Mar 9, 2023

  1. First draft storage layer cli

    Adds a `piker storage` subcmd with a `-d` flag to wipe a particular
    fqsn's time series (both 1s and 60s). Obviously this needs to be
    extended much more but provides a start point.
    goodboy committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    fe0695f View commit details
    Browse the repository at this point in the history
  2. Support startup-config overrides to ahabd super

    With the addition of a new `elastixsearch` docker support in
    #464, adjustments were made
    to container startup sync logic (particularly the `trio` checkpoint
    sleep period - which itself is a hack around a sync client api) which
    caused a regression in upstream startup logic wherein container error
    logs were not being bubbled up correctly causing a silent failure mode:
    
    - `marketstore` container started with corrupt input config
    - `ahabd` super code timed out on startup phase due to a larger log
      polling period, skipped processing startup logs from the container,
      and continued on as though the container was started
    - history client fails on grpc connection with no clear error on why the
      connection failed.
    
    Here we revert to the old poll period (1ms) to avoid any more silent
    failures and further extend supervisor control through a configuration
    override mechanism. To address the underlying design issue, this patch
    adds support for container-endpoint-callbacks to override supervisor
    startup configuration parameters via the 2nd value in their returned
    tuple: the already delivered configuration `dict` value.
    
    The current exposed values include:
        {
            'startup_timeout': 1.0,
            'startup_query_period': 0.001,
            'log_msg_key': 'msg',
        },
    
    This allows for container specific control over the startup-sync query
    period (the hack mentioned above)  as well as the expected log msg key
    and of course the startup timeout.
    goodboy committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    7b196b1 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    959e423 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    8c66f06 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    05b67c2 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    b078a06 View commit details
    Browse the repository at this point in the history
  7. Background docker-container logs processing

    Previously we would make the `ahabd` supervisor-actor sync to docker
    container startup using pseudo-blocking log message processing.
    
    This has issues,
    - we're forced to do a hacky "yield back to `trio`" in order to be
      "fake async" when reading the log stream and further,
    - blocking on a message is fragile and often slow.
    
    Instead, run the log processor in a background task and in the parent
    task poll for the container to be in the client list using a similar
    pseudo-async poll pattern. This allows the super to `Context.started()`
    sooner (when the container is actually registered as "up") and thus
    unblock its (remote) caller faster whilst still doing full log msg
    proxying!
    
    Deatz:
    - adds `Container.cuid: str` a unique container id for logging.
    - correctly proxy through the `loglevel: str` from `pikerd` caller task.
    - shield around `Container.cancel()` in the teardown block and use
      cancel level logging in that method.
    goodboy committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    7694419 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    bb723ab View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    56629b6 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    bfe3ea1 View commit details
    Browse the repository at this point in the history
  11. Start piker.service sub-package

    For now just moves everything that was in `piker._daemon` to a subpkg
    module but a reorg is coming pronto!
    goodboy committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    93c81fa View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    afac553 View commit details
    Browse the repository at this point in the history
  13. Bump mkts timeout to 2s

    goodboy committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    dd87d11 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    b226b67 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    31f2b01 View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    a2d4093 View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    eca048c View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    f95ea19 View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    cec2967 View commit details
    Browse the repository at this point in the history
  20. Attempt to report piker storage -d <fqsn> errors

    Not really sure there's much we can do besides dump Grpc stuff when we
    detect an "error" `str` for the moment..
    
    Either way leave a buncha complaints (como siempre) and do linting
    fixups..
    goodboy committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    441243f View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    6f92c6b View commit details
    Browse the repository at this point in the history
  22. Fix final missed marketstore mod import

    Thanks @esme! XD
    
    Also, do a linter pass and remove a buncha unused references.
    goodboy committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    cda7a54 View commit details
    Browse the repository at this point in the history
  23. Configuration menu
    Copy the full SHA
    fbc12b1 View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    6540c41 View commit details
    Browse the repository at this point in the history
  25. Configuration menu
    Copy the full SHA
    31392af View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    75b7a8b View commit details
    Browse the repository at this point in the history
  27. Add reconnect loop to marketstore startup test

    Due to making ahabd supervisor init more async we need to be more
    tolerant to mkts server startup: the grpc machinery needs to be up
    otherwise a client which connects to early may just hang on requests..
    
    Add a reconnect loop (which might end up getting factored into client
    code too) so that we only block on requests once we know the client
    connection is actually responsive.
    goodboy committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    2014019 View commit details
    Browse the repository at this point in the history
  28. Configuration menu
    Copy the full SHA
    aa36abf View commit details
    Browse the repository at this point in the history
  29. Pull testing config dir from tractor runtime vars

    Provides a more correct solution (particularly for distributed testing)
    to override the `piker` configuration directory by reading the path from
    a specific `tractor._state._runtime_vars` entry that can be provided by
    the test harness.
    
    Also fix some typing and comments.
    goodboy committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    5aaa7f4 View commit details
    Browse the repository at this point in the history
  30. Configuration menu
    Copy the full SHA
    79b0db4 View commit details
    Browse the repository at this point in the history
  31. Configuration menu
    Copy the full SHA
    7cc9911 View commit details
    Browse the repository at this point in the history
  32. Configuration menu
    Copy the full SHA
    9a00c45 View commit details
    Browse the repository at this point in the history
  33. ahabd: Harden cancellation teardown (again XD)

    Needed to move the startup sequence inside the `try:` block to guarantee
    we always do the (now shielded) `.cancel()` call if we get a cancel
    during startup.
    
    Also, support an optional `started_afunc` field in the config if
    backends want to just provide a one-off blocking async func to sync
    container startup. Add a `drop_root_perms: bool` to allow persisting
    sudo perms for testing or dyanmic container spawning purposes.
    goodboy committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    15064d9 View commit details
    Browse the repository at this point in the history
  34. Configuration menu
    Copy the full SHA
    0772b4a View commit details
    Browse the repository at this point in the history
  35. Configuration menu
    Copy the full SHA
    44a3115 View commit details
    Browse the repository at this point in the history
  36. Configuration menu
    Copy the full SHA
    97290fc View commit details
    Browse the repository at this point in the history
  37. Configuration menu
    Copy the full SHA
    8ceaa27 View commit details
    Browse the repository at this point in the history

Commits on Mar 10, 2023

  1. Don't double send enable_modules and debug_mode in kwargs..

    This broke non-disti-mode actor tree spawn / runtime, seemingly because
    the cli entrypoint for a `piker chart` also sends these values down
    through the call stack independently? Pretty sure we don't need to send
    the `enable_modules` from the chart actor anyway.
    goodboy committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    12883c3 View commit details
    Browse the repository at this point in the history