Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automatic offset tracking for stream queues #661

Draft
wants to merge 68 commits into
base: main
Choose a base branch
from
Draft

Automatic offset tracking for stream queues #661

wants to merge 68 commits into from

Commits on Nov 12, 2024

  1. WIP adds automatic offset tracking for streams

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    533ab86 View commit details
    Browse the repository at this point in the history

  2. specs

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    8636ec6 View commit details
    Browse the repository at this point in the history

  3. add specs for edge cases. refactor specs so they are more readable

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    9e6c236 View commit details
    Browse the repository at this point in the history

  4. format

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    cc3718f View commit details
    Browse the repository at this point in the history

  5. refactor, dont init consumer tracking stuff unless needed. resize fil… 

    …e on init to facilitate simple write.
    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    823562e View commit details
    Browse the repository at this point in the history

  6. add spec that checks that only one entry is saved per consumer tag

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    bc291dd View commit details
    Browse the repository at this point in the history

  7. format

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    f0ff66e View commit details
    Browse the repository at this point in the history

  8. lint

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    632934a View commit details
    Browse the repository at this point in the history

  9. add function to remove consumer tags from file

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    92280a2 View commit details
    Browse the repository at this point in the history

  10. spec for removing consumer tags

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    1d5e97d View commit details
    Browse the repository at this point in the history

  11. save length of ctag in file, dont use space as deliminator

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    40cb965 View commit details
    Browse the repository at this point in the history

  12. format

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    3bf4258 View commit details
    Browse the repository at this point in the history

  13. lint

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    9455db1 View commit details
    Browse the repository at this point in the history

  14. remove methods, use instance variables directly. add cleanup function

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    7127e6e View commit details
    Browse the repository at this point in the history

  15. add spec for cleanup

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    e3955ee View commit details
    Browse the repository at this point in the history

  16. raise before updating instance variable 

    Co-authored-by: Carl Hörberg <carl@84codes.com>
    @viktorerlingsson @carlhoerberg
    viktorerlingsson and carlhoerberg committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    50551b4 View commit details
    Browse the repository at this point in the history

  17. refactor 

    Co-authored-by: Carl Hörberg <carl@84codes.com>
    @viktorerlingsson @carlhoerberg
    viktorerlingsson and carlhoerberg committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    94e6db9 View commit details
    Browse the repository at this point in the history

  18. remove comment 

    Co-authored-by: Carl Hörberg <carl@84codes.com>
    @viktorerlingsson @carlhoerberg
    viktorerlingsson and carlhoerberg committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    cab1200 View commit details
    Browse the repository at this point in the history

  19. refactor 

    Co-authored-by: Carl Hörberg <carl@84codes.com>
    @viktorerlingsson @carlhoerberg
    viktorerlingsson and carlhoerberg committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    1606bfa View commit details
    Browse the repository at this point in the history

  20. refactor 

    Co-authored-by: Carl Hörberg <carl@84codes.com>
    @viktorerlingsson @carlhoerberg
    viktorerlingsson and carlhoerberg committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    9fe79b2 View commit details
    Browse the repository at this point in the history

  21. format

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    2f12ad0 View commit details
    Browse the repository at this point in the history

  22. set size to 32768 for offsets file, change path name

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    f047c6e View commit details
    Browse the repository at this point in the history

  23. remove comment

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    7213715 View commit details
    Browse the repository at this point in the history

  24. refactor restore_consumer_offset_positions

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    5f854d8 View commit details
    Browse the repository at this point in the history

  25. remove unused var

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    8c6bcf8 View commit details
    Browse the repository at this point in the history

  26. save_offset_by_consumer_tag -> update_consumer_offset

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    34c7a03 View commit details
    Browse the repository at this point in the history

  27. write_new_ctag_to_file -> store_consumer_offset

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    39f45fe View commit details
    Browse the repository at this point in the history

  28. refactor update_consumer_offset

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    a1c73d7 View commit details
    Browse the repository at this point in the history

  29. add option to get a writeable slice from mfile

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    0657ada View commit details
    Browse the repository at this point in the history

  30. encode directly into mmap. fix bug with pos/size

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    85e1955 View commit details
    Browse the repository at this point in the history

  31. update spec

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    ad69700 View commit details
    Browse the repository at this point in the history

  32. more strict with types

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    51d199a View commit details
    Browse the repository at this point in the history

  33. format

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    b6c079d View commit details
    Browse the repository at this point in the history

  34. lint

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    00b05f0 View commit details
    Browse the repository at this point in the history

  35. lint

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    791c6a7 View commit details
    Browse the repository at this point in the history

  36. String#bytesize 

    Co-authored-by: Carl Hörberg <carl@84codes.com>
    @viktorerlingsson @carlhoerberg
    viktorerlingsson and carlhoerberg committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    ab1e624 View commit details
    Browse the repository at this point in the history

  37. refactor store_consumer_offset and restore_consumer_offset_positions

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    dbe06f1 View commit details
    Browse the repository at this point in the history

  38. refactor cleanup_consumer_offsets

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    b608a8f View commit details
    Browse the repository at this point in the history

  39. replace offset file instead of delete

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    58c48d4 View commit details
    Browse the repository at this point in the history

  40. lint

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    3413949 View commit details
    Browse the repository at this point in the history

  41. lint

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    f6340f2 View commit details
    Browse the repository at this point in the history

  42. dont track offsets when consumer tag is generated

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    48849af View commit details
    Browse the repository at this point in the history

  43. remove unused code

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    5b071c9 View commit details
    Browse the repository at this point in the history

  44. cleanup consumer offsets when dropping overflow

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    f9bc466 View commit details
    Browse the repository at this point in the history

  45. format

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    2b31cbd View commit details
    Browse the repository at this point in the history

  46. lint

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    1619e35 View commit details
    Browse the repository at this point in the history

  47. handle large messages causing first segment to be empty

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    e4361c0 View commit details
    Browse the repository at this point in the history

  48. cleanup spec

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    e2c9c44 View commit details
    Browse the repository at this point in the history

  49. add option to use broker tracking when x-stream-offset is set by usin… 

    …g x-stream-use-automatic-offset
    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    87dd311 View commit details
    Browse the repository at this point in the history

  50. add spec

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    28bdf4c View commit details
    Browse the repository at this point in the history

  51. no need to truncate mfile, it's being deleted 

    Co-authored-by: Carl Hörberg <carl@84codes.com>
    @viktorerlingsson @carlhoerberg
    viktorerlingsson and carlhoerberg committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    1e0a92b View commit details
    Browse the repository at this point in the history

  52. x-stream-use-automatic-offset -> x-stream-automatic-offset-tracking

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    b170d16 View commit details
    Browse the repository at this point in the history

  53. implement rename in mfile

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    6dcd056 View commit details
    Browse the repository at this point in the history

  54. expand consumer offsets file if full

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    23036e1 View commit details
    Browse the repository at this point in the history

  55. remove unused code

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    526b81e View commit details
    Browse the repository at this point in the history

  56. use LittleEndian 

    Co-authored-by: Carl Hörberg <carl@84codes.com>
    @viktorerlingsson @carlhoerberg
    viktorerlingsson and carlhoerberg committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    11cabda View commit details
    Browse the repository at this point in the history

  57. remove instance variables

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    7d48d7e View commit details
    Browse the repository at this point in the history

  58. use old_consumer_offsets.path

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    265abcd View commit details
    Browse the repository at this point in the history

  59. start reading at pos=4 after IndexError in offset_at

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    e4b2951 View commit details
    Browse the repository at this point in the history

  60. use queue_data_dir

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    37a446f View commit details
    Browse the repository at this point in the history

  61. update specs to start amqp servers where needed

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    9e3280f View commit details
    Browse the repository at this point in the history

  62. ameba:disable Metrics/CyclomaticComplexity for find_offset

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    a0bf581 View commit details
    Browse the repository at this point in the history

  63. include tag size prefix byte in consumer_offset_file_full? 

    Co-authored-by: Carl Hörberg <carl@84codes.com>
    @viktorerlingsson @carlhoerberg
    viktorerlingsson and carlhoerberg committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    601217e View commit details
    Browse the repository at this point in the history

  64. only append consumer offsets file, compact when full, expand if still… 

    … full. Use config.instance.segment_size
    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    a526419 View commit details
    Browse the repository at this point in the history

  65. set capacity of consumer offsets file to 1000*current size when compa… 

    …cting
    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    e1a0bf8 View commit details
    Browse the repository at this point in the history

  66. replicate consumer offsets file. remove unused code

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    cf1cca7 View commit details
    Browse the repository at this point in the history

  67. fixes to match changes from main

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    f7aad09 View commit details
    Browse the repository at this point in the history

  68. lint

    @viktorerlingsson
    viktorerlingsson committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    cce2560 View commit details
    Browse the repository at this point in the history