Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Secondary locations heatmaps, downloads & tests #5745

Closed
wants to merge 29 commits into from
Closed

[WIP] Secondary locations heatmaps, downloads & tests #5745

wants to merge 29 commits into from

Commits on Nov 9, 2023

  1. neon_local: remote storage by default & enable multiple pageservers in 

    init
    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    9fd901b View commit details
    Browse the repository at this point in the history

  2. control_plane: add /inspect to attachment service

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    c7e6668 View commit details
    Browse the repository at this point in the history

  3. control_plane: implement cargo neon migrate

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    c27c7d8 View commit details
    Browse the repository at this point in the history

  4. neon_local: don't use default remote storage if user specified it on CLI

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    28f752f View commit details
    Browse the repository at this point in the history

  5. Use the same local_fs path as tests

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    11e9916 View commit details
    Browse the repository at this point in the history

  6. tests: explicitly disable remote storage if it is None

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    b698bc9 View commit details
    Browse the repository at this point in the history

  7. control_plane: use pageserver/ sub path for local_fs

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    497b09b View commit details
    Browse the repository at this point in the history

  8. pageserver: add secondary-mode metrics

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    6f0a225 View commit details
    Browse the repository at this point in the history

  9. pageserver: add TenantConf::enable_heatmap (default false)

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    fc3c7aa View commit details
    Browse the repository at this point in the history

  10. pageserver: add secondary downloader & heatmaps

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    c60ec31 View commit details
    Browse the repository at this point in the history

  11. pageserver: TenantManager support for SecondaryTenant

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    06e0370 View commit details
    Browse the repository at this point in the history

  12. Add Timeline::generate_heatmap, remote client heatmap upload

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    c7dd9d9 View commit details
    Browse the repository at this point in the history

  13. pageserver: launch tasks for secondary mode

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    80e18ac View commit details
    Browse the repository at this point in the history

  14. Refactor Workload into shared location

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    56fe8ce View commit details
    Browse the repository at this point in the history

  15. tests: refactor a path helper

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    30d9830 View commit details
    Browse the repository at this point in the history

  16. tests: add helpers for location_conf

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    ce29d15 View commit details
    Browse the repository at this point in the history

  17. tests: add tests for secondary mode & multi attach

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    33a66b9 View commit details
    Browse the repository at this point in the history

  18. pageserver: implement flush on location conf update

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    afc8315 View commit details
    Browse the repository at this point in the history

  19. pageserver/http: add testing routes for secondary mode

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    8135cd7 View commit details
    Browse the repository at this point in the history

  20. pageserver: create timelines/ dir when configuring secondary location

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    8506008 View commit details
    Browse the repository at this point in the history

  21. pageserver: add Layer::for_secondary

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    d9e2807 View commit details
    Browse the repository at this point in the history

  22. pageserver: pass TenantManager into disk usage eviction task

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    b55e295 View commit details
    Browse the repository at this point in the history

  23. pageserver: include secondary tenants in disk usage eviction

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    028b57a View commit details
    Browse the repository at this point in the history

  24. pageserver: add debug_assert_current_span_has_tenant_id

    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    c5f7c6b View commit details
    Browse the repository at this point in the history

  25. tests: set a backoff_factor for test HTTP retries 

    Previously we were rapid-retrying, which worked when
doing local loads for activating tenants, but gives us
very little time for tenants that are actually taking
some time to activate, e.g. in tests that use fault
injection for remote storage, or lots of layers.
    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    c4bd100 View commit details
    Browse the repository at this point in the history

  26. Merge branch 'jcsp/neon-local-migration' into jcsp/secondary-location… 

    …s-pt3
    @jcsp
    jcsp committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    9c52a30 View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2023

  1. Use secondary mode bits in neon_local migration

    @jcsp
    jcsp committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    6901bf4 View commit details
    Browse the repository at this point in the history

  2. f http routes

    @jcsp
    jcsp committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    ebf151f View commit details
    Browse the repository at this point in the history

  3. pageserver: error type for collect_keyspace

    @jcsp
    jcsp committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    3107010 View commit details
    Browse the repository at this point in the history