Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
109 commits
Select commit Hold shift + click to select a range
b42b3f9
add approx. transactional api
blindspotbounty Jul 24, 2023
f5bfbe2
add state for initalized transactions
blindspotbounty Jul 25, 2023
b2c1619
Merge branch 'main' into feature/sc-2764/gsoc-support-for-transactions
blindspotbounty Jul 26, 2023
c870864
introduce statistics for producer
blindspotbounty Jul 26, 2023
4acd979
experiments
blindspotbounty Jul 28, 2023
4f9520f
Merge remote-tracking branch 'origin/main' into feature/sc-2764/gsoc-…
blindspotbounty Jul 28, 2023
7da4901
address some review comments
blindspotbounty Jul 31, 2023
ce702f3
use shared config with producer
blindspotbounty Jul 31, 2023
633773f
Merge remote-tracking branch 'origin/main' into feature/sc-1976/gsoc-…
blindspotbounty Jul 31, 2023
d9887b9
add statistics to new consumer with events
blindspotbounty Jul 31, 2023
e5f0483
Merge remote-tracking branch 'origin/main' into feature/sc-1976/gsoc-…
blindspotbounty Aug 10, 2023
d55a7fd
fix some artefacts
blindspotbounty Aug 10, 2023
8b4525b
adjust to KeyRefreshAttempts
blindspotbounty Aug 10, 2023
8118757
Merge remote-tracking branch 'origin/main' into feature/sc-2764/gsoc-…
blindspotbounty Aug 10, 2023
6501aaa
remove merge artifact
blindspotbounty Aug 10, 2023
1b32437
Merge branch 'feature/sc-2764/gsoc-support-for-transactions' into sta…
blindspotbounty Aug 10, 2023
f2a1461
tmp merge
blindspotbounty Aug 10, 2023
b5df926
tmp
blindspotbounty Aug 10, 2023
c878c2d
add seek/assign
blindspotbounty Aug 10, 2023
87eedae
adoptation
blindspotbounty Aug 10, 2023
b53fa16
add some helpers
blindspotbounty Aug 10, 2023
91a49e8
enable rebalance
blindspotbounty Aug 11, 2023
447c1c0
tmp
blindspotbounty Aug 11, 2023
a8cf21d
add return in assign
blindspotbounty Aug 14, 2023
cfd44f0
return offset.store
blindspotbounty Aug 14, 2023
df8c9ce
quick hack to avoid deadlock
blindspotbounty Aug 14, 2023
26fafb2
add some experimental code
blindspotbounty Aug 17, 2023
7c929be
revert auto.offset.store=false
blindspotbounty Aug 17, 2023
62912ae
revert class->struct
blindspotbounty Aug 18, 2023
b012c52
better interval handling
blindspotbounty Aug 23, 2023
8f7b0c6
add flush
blindspotbounty Aug 23, 2023
f801132
return true
blindspotbounty Aug 23, 2023
8eafb28
sources/test
blindspotbounty Aug 29, 2023
6120201
store eof offset without +1
blindspotbounty Aug 29, 2023
ce1d711
adpative poll interval
blindspotbounty Aug 31, 2023
e0bb63d
backpressure experiments
blindspotbounty Sep 13, 2023
eda7ca9
backpressure implemented
blindspotbounty Oct 5, 2023
91a1ce3
forgotten cleanup commit
blindspotbounty Oct 5, 2023
68f35db
chore(patch): [sc-2205] Switch librdkafka to our fork
axelandersson Oct 6, 2023
a2c5d4e
chore(patch): [sc-2205] Depend on libcurl/libsasl2 only on macOS and …
axelandersson Oct 6, 2023
44523be
Fix Package.swift syntax
axelandersson Oct 6, 2023
d5ac008
chore: [sc-2205] Update librdkafka
axelandersson Oct 6, 2023
51dcaa7
ci: add semantic release
ordo-ci Oct 6, 2023
210f0c1
+flush
blindspotbounty Oct 6, 2023
99ca0a2
logs for transactions
blindspotbounty Oct 6, 2023
50526ea
info->debug
blindspotbounty Oct 6, 2023
87fc536
ci(patch): bump to macos14
ordo-ci Oct 6, 2023
79dc8d2
ci(patch): fix semantic release
ordo-ci Oct 6, 2023
9069a0f
chore(minor): bump version
ordo-ci Oct 6, 2023
32ba68e
chore(minor): bump version
ordo-ci Oct 6, 2023
783298e
chore(minor): bump version
ordo-ci Oct 6, 2023
36e76d8
chore(minor): bump version
ordo-ci Oct 6, 2023
cbc95c0
chore(minor): bump version
ordo-ci Oct 6, 2023
bc8cb36
chore(minor): bump version
ordo-ci Oct 6, 2023
1c47a81
chore(minor): bump version
ordo-ci Oct 6, 2023
d23a604
chore(minor): bump version
ordo-ci Oct 6, 2023
c37fa9f
chore(minor): bump version
ordo-ci Oct 9, 2023
9404d1a
Merge pull request #7 from ordo-one/bug/sc-3570/with-10k-batch-for-tr…
blindspotbounty Oct 9, 2023
4ad373f
feat(minor): [sc-2762] add rebalance strategy support
blindspotbounty Oct 31, 2023
6c05ebc
Merge pull request #9 from ordo-one/feature/sc-2762/gsoc-rebalance-ca…
blindspotbounty Oct 31, 2023
ca6bd61
fix(patch): [sc-4693] throw error after catch to ensure correct error…
blindspotbounty Nov 2, 2023
a5cfc8f
Merge pull request #10 from ordo-one/bug/sc-4693/another-augmenter-st…
blindspotbounty Nov 2, 2023
42c2972
Revert "Ship with BoringSSL (#135)"
FranzBusch Nov 3, 2023
f40705e
Merge remote-tracking branch 'swift-server/main' into feature/sc-4740…
blindspotbounty Nov 3, 2023
a30953d
Merge commit '42c29720e8f0d895e452498046f5a9542572b08c' into feature/…
blindspotbounty Nov 3, 2023
61f982f
define rebalance as callback
blindspotbounty Nov 3, 2023
f84a760
add patch for polls
blindspotbounty Nov 6, 2023
a835906
add comment for future
blindspotbounty Nov 6, 2023
695cb7b
add missed graceful shutdown
blindspotbounty Nov 6, 2023
38b6121
Merge pull request #12 from ordo-one/feature/sc-4740/kafka-rebase-on-…
blindspotbounty Nov 7, 2023
de7534a
fix(patch): [sc-4740] add additional failure as success #13
blindspotbounty Nov 7, 2023
dbc8008
fix(patch): [sc-4740] take event source from state machine rather tha…
blindspotbounty Nov 8, 2023
ab183f4
chore(major): [sc-4804] swift kafka wrap rebalance related blocking (…
blindspotbounty Nov 10, 2023
731ac14
fix(patch): [sc-4897] move rebalance callback storage to rd kafka cli…
blindspotbounty Nov 14, 2023
209d314
ci: 🔄 synced local '.github/workflows/' with remote 'workflows/swift'
Nov 20, 2023
1b21544
ci: 🔄 created local '.github/actions/' from remote 'actions'
Nov 20, 2023
c56d842
ci: 🔄 created local '.swiftlint.yml' from remote 'dotfiles/swiftlint.…
Nov 20, 2023
397a414
ci: 🔄 synced local '.swiftformat' with remote 'dotfiles/swiftformat'
Nov 20, 2023
a457c3a
ci: 🔄 synced local '.gitignore' with remote 'dotfiles/gitignore'
Nov 20, 2023
9d723fb
ci: 🔄 created local '.swift-version' from remote 'dotfiles/swift-vers…
Nov 20, 2023
ba8991a
ci: 🔄 created local '.github/pull_request_template.md' from remote 'w…
Nov 20, 2023
8f7113a
ci: 🔄 synced local '.github/workflows/' with remote 'workflows/swift'
Nov 20, 2023
aeddca2
ci: 🔄 synced local '.github/workflows/' with remote 'workflows/swift'
Nov 20, 2023
55bab55
feat(minor): support calculation of partition from key [sc-2817]
dimlio Nov 20, 2023
7e0737c
feat(minor): support calculation Kafka partition from key [sc-2817]
dimlio Nov 22, 2023
dbe6019
revert release to ordoalpha swift-kafka-client
ordo-ci Nov 22, 2023
894c0a6
chore(minor): [sc-2205] to latest librdkafka & add a define for RAND_…
blindspotbounty Nov 28, 2023
a47eb0f
Revert "Revert "Ship with BoringSSL (#135)"" (#22)
blindspotbounty Nov 29, 2023
18569c6
feat(minor): [sc-2373] provide partition eof based on configuration (…
blindspotbounty Dec 1, 2023
01c4514
chore(minor): [sc-2373] allow to get metadata from broker (#23)
blindspotbounty Dec 4, 2023
b55fc2e
exit from consumer batch loop when no more messages left (#153) (#24)
blindspotbounty Dec 7, 2023
8b708c7
exit from consumer batch loop when no more messages left (#153) (#25)
blindspotbounty Dec 12, 2023
ac02f09
fix(patch): [sc-xxxx] dummy commit (#26)
blindspotbounty Dec 14, 2023
6bec4a7
chore(patch): [sc-5640] remove async stream: wrap `rd_kafka_consumer_…
blindspotbounty Dec 22, 2023
8d6d7ad
fix(patch): [sc-xxx] update dependency (#28)
blindspotbounty Dec 22, 2023
1de01dd
fix(patch): [sc-xxx] dial with rebalance on shutdown (#29)
blindspotbounty Dec 22, 2023
3348cbd
fix(patch): [sc-xxx] finish source sequence even when excepetion thro…
blindspotbounty Jan 29, 2024
b1253c3
chore(patch): [sc-7382] use patch with fixed double free (#31)
blindspotbounty Mar 5, 2024
b73a399
chore(patch): [sc-8379] use returned error (#32)
blindspotbounty Mar 26, 2024
e6dfe2c
fix(minor): [sc-8842] Lower requirements for consumer state machine (…
blindspotbounty Apr 11, 2024
1371c56
chore(patch): [sc-11865] allow to provide several partitions for assi…
blindspotbounty Jul 12, 2024
54378b6
ci: update semantic release workflow
ordo-ci Jul 12, 2024
81a4b7f
chore(patch): re-release
ordo-ci Aug 6, 2024
aa8fdfa
chore(patch): re-release
ordo-ci Aug 6, 2024
52ceca1
chore(patch): re-release
ordo-ci Aug 6, 2024
1f2076f
chore(patch): [sc-12737] update from upstream (#35)
blindspotbounty Aug 6, 2024
b04bb7e
chore(patch): [sc-10850] add method to create topic with specified na…
blindspotbounty Aug 6, 2024
20c3967
fix(minor): [sc-9450] Add possibility to fetch list of consumer groups.
ser-0xff Aug 7, 2024
a0714f2
Propagate most group info.
ser-0xff Aug 7, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .github/actions/consul-start/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
name: "Start Consul"
runs:
using: "composite"
steps:
- uses: nahsi/setup-hashi-tool@v1
if: github.repository == 'ordo-one/package-consul' || github.repository == 'ordo-one/package-distributed-system'
with:
name: consul
- name: Start consul
if: github.repository == 'ordo-one/package-consul' || github.repository == 'ordo-one/package-distributed-system'
shell: bash
run: |
consul agent -dev -log-level=warn &
13 changes: 13 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
## Description

Include a summary of the change and which issue is fixed. Please also include relevant motivation and context.

## How Has This Been Tested?

Please describe the tests that you ran to verify your changes.

## Minimal checklist:

- [ ] I have performed a self-review of my own code
- [ ] I have added `DocC` code-level documentation for any public interfaces exported by the package
- [ ] I have added unit and/or integration tests that prove my fix is effective or that my feature works
65 changes: 65 additions & 0 deletions .github/workflows/lint-pr.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
name: "PR title validation"

on:
pull_request_target:
types:
- opened
- edited
- synchronize

jobs:
main:
name: Validate PR title
# runs-on: [self-hosted, ubuntu-latest]
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: amannn/action-semantic-pull-request@v5
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
# Configure which types are allowed.
# Default: https://github.com/commitizen/conventional-commit-types
types: |
fix
feat
docs
style
refactor
perf
test
build
ci
chore
revert
# Configure which scopes are allowed.
scopes: |
patch
hotfix
minor
major
# Configure that a scope must always be provided.
requireScope: false
# Configure which scopes are disallowed in PR titles. For instance by setting
# the value below, `chore(release): ...` and `ci(e2e,release): ...` will be rejected.
disallowScopes: |
release
# Configure additional validation for the subject based on a regex.
# This example ensures the subject doesn't start with an uppercase character.
#subjectPattern: ^.*\[sc-[0-9]+\].*$
#subjectPatternError: |
# The pull request title should contain Shortcut case number like '[sc-123]'
# If the PR contains one of these labels, the validation is skipped.
# Multiple labels can be separated by newlines.
# If you want to rerun the validation when labels change, you might want
# to use the `labeled` and `unlabeled` event triggers in your workflow.
ignoreLabels: |
bot
ignore-semantic-pull-request
# For work-in-progress PRs you can typically use draft pull requests
# from GitHub. However, private repositories on the free plan don't have
# this option and therefore this action allows you to opt-in to using the
# special "[WIP]" prefix to indicate this state. This will avoid the
# validation of the PR title and the pull request checks remain pending.
# Note that a second check will be reported if this is enabled.
wip: true
66 changes: 66 additions & 0 deletions .github/workflows/semantic-release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
name: Semantic Release

on:
workflow_dispatch:
push:
branches: [ main, next ]

jobs:
semantic-release:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Semantic Release Config
run: |
echo "
branches:
- main
- name: 'next'
prerelease: 'ordoalpha'
preset: 'conventionalcommits'
tagFormat: '\${version}'
plugins:
- - '@semantic-release/commit-analyzer'
- releaseRules:
- breaking: true
release: major
- type: build
release: patch
- type: chore
release: false
- type: feat
release: false
- type: fix
release: false

- scope: 'no-release'
release: false
- scope: 'hotfix'
release: patch
- scope: 'patch'
release: patch
- scope: 'minor'
release: minor
- scope: 'major'
release: major
- - '@semantic-release/release-notes-generator'
- - '@semantic-release/github'
- successComment: false
failTitle: false" > .releaserc.yml

- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'

- name: Install semantic-release
run: |
npm install semantic-release@v24 conventional-changelog-conventionalcommits@v8 -D
npm list
- name: Release
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: npx semantic-release
91 changes: 91 additions & 0 deletions .github/workflows/swift-benchmark-delta.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
name: Benchmark PR vs main

on:
workflow_dispatch:
pull_request:
branches: [ main ]

jobs:
benchmark-delta:

runs-on: ${{ matrix.os }}
continue-on-error: true

strategy:
matrix:
#os: [[Linux, benchmark-swift-latest, self-hosted]]
os: [ubuntu-latest]

steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Homebrew Mac
if: ${{ runner.os == 'Macos' }}
run: |
echo "/opt/homebrew/bin:/usr/local/bin" >> $GITHUB_PATH
brew install jemalloc

- name: Ubuntu deps
if: ${{ runner.os == 'Linux' }}
run: |
sudo apt-get install -y libjemalloc-dev libsasl2-dev libcurl4-openssl-dev

- name: Start consul
uses: ./.github/actions/consul-start

- name: Git URL token override and misc
run: |
#git config --global url."https://ordo-ci:${{ secrets.CI_MACHINE_PAT }}@github.com".insteadOf "https://github.com"
#/usr/bin/ordo-performance
[ -d Benchmarks ] && echo "hasBenchmark=1" >> $GITHUB_ENV
echo "/opt/homebrew/bin:/usr/local/bin" >> $GITHUB_PATH
- name: Run benchmarks for PR branch
if: ${{ env.hasBenchmark == '1' }}
run: |
swift package --allow-writing-to-directory .benchmarkBaselines/ benchmark baseline update pull_request --no-progress --quiet
- name: Switch to branch 'main'
if: ${{ env.hasBenchmark == '1' }}
run: |
git stash
git checkout main
- name: Run benchmarks for branch 'main'
if: ${{ env.hasBenchmark == '1' }}
run: |
swift package --allow-writing-to-directory .benchmarkBaselines/ benchmark baseline update main --no-progress --quiet
- name: Compare PR and main
if: ${{ env.hasBenchmark == '1' }}
id: benchmark
run: |
echo $(date) >> $GITHUB_STEP_SUMMARY
echo "exitStatus=1" >> $GITHUB_ENV
swift package benchmark baseline check main pull_request --format markdown >> $GITHUB_STEP_SUMMARY
echo "exitStatus=0" >> $GITHUB_ENV
continue-on-error: true
- if: ${{ env.exitStatus == '0' }}
name: Pull request comment text success
id: prtestsuccess
run: |
echo 'PRTEST<<EOF' >> $GITHUB_ENV
echo "[Pull request benchmark comparison [${{ matrix.os }}] with 'main' run at $(date -Iseconds)](https://github.com/ordo-one/${{ github.event.repository.name }}/actions/runs/${{ github.run_id }})" >> $GITHUB_ENV
echo 'EOF' >> $GITHUB_ENV
- if: ${{ env.exitStatus == '1' }}
name: Pull request comment text failure
id: prtestfailure
run: |
echo 'PRTEST<<EOF' >> $GITHUB_ENV
echo "[Pull request benchmark comparison [${{ matrix.os }}] with 'main' run at $(date -Iseconds)](https://github.com/ordo-one/${{ github.event.repository.name }}/actions/runs/${{ github.run_id }})" >> $GITHUB_ENV
echo "_Pull request had performance regressions_" >> $GITHUB_ENV
echo 'EOF' >> $GITHUB_ENV
- name: Comment PR
if: ${{ env.hasBenchmark == '1' }}
uses: thollander/actions-comment-pull-request@v1
with:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
message: ${{ env.PRTEST }}
comment_includes: "Pull request benchmark comparison [${{ matrix.os }}] with"
- name: Exit with correct status
run: |
#/usr/bin/ordo-performance powersave
exit ${{ env.exitStatus }}
27 changes: 27 additions & 0 deletions .github/workflows/swift-check-api-breaks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
name: Swift check API breaks

on:
workflow_dispatch:
pull_request:
types: [opened, synchronize]

jobs:
analyze-api-breakage:

runs-on: [ubuntu-latest]
timeout-minutes: 30

steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
submodules: 'true'

- name: Ubuntu deps
if: ${{ runner.os == 'Linux' }}
run: |
sudo apt-get install -y libjemalloc-dev libsasl2-dev libcurl4-openssl-dev
- name: Build
run: swift build
- name: Analyze API breakage
run: swift package diagnose-api-breaking-changes origin/next
48 changes: 48 additions & 0 deletions .github/workflows/swift-code-coverage.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
name: Swift code coverage

on:
workflow_dispatch:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
test-code-coverage:
runs-on: [ubuntu-22.04]
timeout-minutes: 60
steps:
- uses: actions/checkout@v3
with:
submodules: 'true'

- name: Ubuntu deps
if: ${{ runner.os == 'Linux' }}
run: |
sudo apt-get install -y libjemalloc-dev libsasl2-dev libcurl4-openssl-dev llvm-15

- name: Start consul
uses: ./.github/actions/consul-start

- name: Run tests
continue-on-error: true
run: |
[ -d Tests ] && swift test --parallel --enable-code-coverage

- name: Export code coverage
run: |
xctest_binary=".build/debug/${{ github.event.repository.name }}PackageTests.xctest"
if [ ! -f ${xctest_binary} ]; then
xctest_binary=$(find .build/debug/ -type f -name "*.xctest" | tail -1)
echo "Will llvm-cov '${xctest_binary}'"
fi

if [ -f ${xctest_binary} ]; then
llvm-cov-15 export -format="lcov" ${xctest_binary} -instr-profile .build/debug/codecov/default.profdata > info.lcov
fi

- name: Upload codecov
uses: codecov/codecov-action@v2
with:
token: ${{ secrets.CODECOV_REPO_TOKEN }}
files: info.lcov
fail_ci_if_error: true
20 changes: 20 additions & 0 deletions .github/workflows/swift-lint.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
name: Swift lint

on:
workflow_dispatch:
pull_request:
paths:
- '.github/workflows/swiftlint.yml'
- '.swiftlint.yml'
- '**/*.swift'

jobs:
SwiftLint:
timeout-minutes: 60
runs-on: [ubuntu-latest]
steps:
- uses: actions/checkout@v3
- name: GitHub Action for SwiftLint with --strict
uses: norio-nomura/action-swiftlint@3.2.1
with:
args: --strict
44 changes: 44 additions & 0 deletions .github/workflows/swift-linux-build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
name: Swift Linux build

on:
workflow_dispatch:
push:
branches: [ main ]
pull_request:
branches: [ main, next ]

jobs:
build-linux:
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest]
swift: ["5.9"]


runs-on: ${{ matrix.os }}
steps:
- uses: fwal/setup-swift@v1
if: ${{ false }}
with:
swift-version: ${{ matrix.swift }}

- uses: actions/checkout@v3
with:
submodules: 'true'

- name: Start consul
uses: ./.github/actions/consul-start

- name: Ubuntu deps
if: ${{ runner.os == 'Linux' }}
run: |
sudo apt-get install -y libjemalloc-dev libsasl2-dev libcurl4-openssl-dev

- name: Swift version
run: swift --version

- name: Run tests
continue-on-error: true
run: docker-compose -f docker/docker-compose.yaml -f docker/docker-compose.2204.59.yaml run test
Loading