chore: make usan asan optional and enable them on CI #2631

kostasrim · 2024-02-20T12:42:53Z

enable usan asan for our unit tests on the CI build
make usan and asan flags optional

kostasrim · 2024-02-20T12:43:06Z

shall I also enable them for the regression tests?

adiholden · 2024-02-29T09:29:21Z

.github/workflows/daily-sanitizers.yml

+
+jobs:
+  build:
+    runs-on: [self-hosted, linux, ARM64]


why do you run it only on the self-hosted machine?

matrix: # Test of these containers container: ["ubuntu-dev:20"] build-type: [Debug runner: [ubuntu-latest, [self-hosted, linux, ARM64]] runs-on: ${{ matrix.runner }}

Because the gh runners run out of memory...

check this out - https://docs.github.com/en/actions/using-github-hosted-runners/about-larger-runners/about-larger-runners

I am aware but really good mention. The reason I did not pursue this is because of the costs of these runners. Normal runners for open source projects get some free time (I think) and then you pay. Whereas with this, you pay every time. As we already pay for the AWS runner I don't see a reason to introduce an extra billing cycle nor can I estimate if it's actually cheaper to run it on gh larger runners 🤷

Ok. we are getting OOM when we are building right? can you use ninja -j4 and see if it helps?

I can try but is it really worh it? We still get an extra failure

what do you mean we get extra failure?

oh you mean to use ubuntu latest with the normal runners? Sure I can try :)

https://github.com/dragonflydb/dragonfly/actions/runs/8098506590/job/22132072377#step:7:973

Failed no space left again

adiholden · 2024-02-29T09:30:56Z

.github/workflows/daily-sanitizers.yml

+    strategy:
+      matrix:
+        # Test of these containers
+        container: ["ubuntu-dev:22"]


so if both tests fail also on ubuntu 22.04 why do we run them on this ubuntu verstion and not 20.04
I thought that running them on ubuntu 22.04 will make them pass

I thought that running them on ubuntu 22.04 will make them pass

We still have a false positive on 20.04 complaining about the mpmc queue on 20:04 (it's the bug that gets away once you update on gcc-11). I was wrong though on the other two issues apparently (the exception thrown in the parser and the hash_table_access) :(

All in all I rather use 22.04 instead of 20.04 mostly because the second is one year away of its end of life and at least one of the false positives goes away by using 22.04. It's still a win, even though a minor one.

I am fine running whatever ubuntu version we need for them to pass if such exists. we can still release via ubuntu 20

adiholden · 2024-02-29T09:35:22Z

.github/workflows/daily-sanitizers.yml

+            ctest -V -L DFLY -E "^(search_family|multi)_test"
+
+      - name: PostFail
+        if: failure()


we need to send notification on failure otherwize no one will see this failed

adiholden · 2024-02-29T09:54:45Z

.github/workflows/daily-sanitizers.yml

+      - name: Test
+        run: |
+            cd $GITHUB_WORKSPACE/build
+            ctest -V -L DFLY -E "^(search_family|multi)_test"


we have 2 test cases that fail right? not the whole search family / multi test
if this is the case I would disable the specif tests that fail
if there is no way to do this with ctest args than add in the test code for the failing tests ifdef WITH_ASAN skip

I did. There is a downside to this though. The problem happens on arm only, so the tests become disabled on x86 as well. I don't expect this to be of high friction so it's fine. Besides I do want a long term solution for this as I would like to run the sanitizers as part of every PR.

oh I used #ifndef because cmake labels are specific to a test and not a test case :(

adiholden · 2024-02-29T14:12:36Z

.github/workflows/daily-sanitizers.yml

+        if: failure()
+        run: |
+          job_link="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
+          message="Regression tests failed.\\n The commit is: ${{github.sha}}.\\n Job Link: ${job_link}\\n"


its not regression tests:) please modify the message

adiholden · 2024-02-29T14:13:40Z

.github/workflows/daily-sanitizers.yml

+      - name: Test
+        run: |
+            cd $GITHUB_WORKSPACE/build
+            #ctest -V -L DFLY


remove the comment and remove the below line

I am testing! HODL!

…nfly ( v1.15.1 → v1.16.0 ) (#3354) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [docker.dragonflydb.io/dragonflydb/dragonfly](https://togithub.com/dragonflydb/dragonfly) | minor | `v1.15.1` -> `v1.16.0` | --- ### Release Notes <details> <summary>dragonflydb/dragonfly (docker.dragonflydb.io/dragonflydb/dragonfly)</summary> ### [`v1.16.0`](https://togithub.com/dragonflydb/dragonfly/releases/tag/v1.16.0) [Compare Source](https://togithub.com/dragonflydb/dragonfly/compare/v1.15.1...v1.16.0) ##### Dragonfly v1.16.0 Our spring release. We are getting closer to 2.0 with some very exciting features ahead. Stay tuned! Some prominent changes include: - Improved memory accounting of client connections ([#2710](https://togithub.com/dragonflydb/dragonfly/issues/2710) [#2755](https://togithub.com/dragonflydb/dragonfly/issues/2755) and [#2692](https://togithub.com/dragonflydb/dragonfly/issues/2692) ) - FT.AGGREGATE call ([#2413](https://togithub.com/dragonflydb/dragonfly/issues/2413)) - Properly handle and replicate Memcache flags ([#2787](https://togithub.com/dragonflydb/dragonfly/issues/2787) [#2807](https://togithub.com/dragonflydb/dragonfly/issues/2807)) - Intoduce BF.AGGREGATE BD.(M)ADD and BF.(M)EXISTS methods ([#2801](https://togithub.com/dragonflydb/dragonfly/issues/2801)). Note, that it does not work with snapshots and replication yet. - Dragonfly builds natively on MacOS. We would love some help with extending the release pipeline to create a proper macos binary. - Following the requests from the Edge developers community, we added a basic HTTP API support! Try running Dragonfly with: `--expose_http_api` flag and then call `curl -X POST -d '["ping"]' localhost:6379/api`. We will follow up with more extensive docs later this month. - Lots of stability fixes, especially around Sidekiq and BullMQ workloads. ##### What's Changed - chore: make usan asan optional and enable them on CI by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2631 - fix: missing manual trigger for daily sanitizers by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2682 - bug(tiering): fix overflow in page offset calculation and wrong hash offset calculation by [@theyueli](https://togithub.com/theyueli) in [dragonflydb/dragonfly#2683 - Chore: Fixed Docker Health Check by [@manojks1999](https://togithub.com/manojks1999) in [dragonflydb/dragonfly#2659 - chore: Increase disk space in the coverage runs by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2684 - fix(flushall): Decommit memory after releasing tables. by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2691 - feat(server): Account for serializer's temporary buffer size by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2689 - refactor: remove FULL-SYNC-CUT cmd [#2687](https://togithub.com/dragonflydb/dragonfly/issues/2687) by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#2688 - chore: add malloc-based stats and decommit by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2692 - feat(cluster): automatic slot migration finalization [#2697](https://togithub.com/dragonflydb/dragonfly/issues/2697) by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#2698 - Basic FT.AGGREGATE by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2413 - chore: little transaction cleanup by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2608 - fix(channel store): add acquire/release pair in fast update path by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2704 - chore: add ubuntu22 devcontainer by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2700 - feat(cluster): Add `--cluster_id` flag by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2695 - feat(server): Use mimalloc in SSL calls by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2710 - chore: Pull helio with new BlockingCounter by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2711 - chore(transaction): Simplify PollExecution by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2712 - chore(transaction): Don't call GetLocalMask from blocking controller by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2715 - chore: improve compatibility of EXPIRE functions with Redis by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2696 - chore: disable flaky fuzzy migration test by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2716 - chore: update sanitizers workflow by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2686 - chore: Use c-ares for resolving hosts in `ProtocolClient` by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2719 - Remove check-fail in ExpireIfNeeded and introduce DFLY LOAD by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2699 - chore: Record cmd stat from invoke by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2720 - fix(transaction): nullptr access on non transactional commands by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2724 - fix(BgSave): async from sync by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2702 - chore: remove core/fibers by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2723 - fix(transaction): Replace with armed sync point by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2708 - feat(json): Deserialize ReJSON format by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2725 - feat: add flag masteruser by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2693 - refactor: block list refactoring [#2580](https://togithub.com/dragonflydb/dragonfly/issues/2580) by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#2732 - chore: fix DeduceExecMode by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2733 - fix(cluster): Reply with correct `\n` / `\r\n` from `CLUSTER` sub cmd by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2731 - chore: Introduce fiber stack allocator by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2730 - fix(cluster): Save replica ID per replica by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2735 - fix(ssl): Proper cleanup by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2742 - chore: add skeleton files for flat_dfs code by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2738 - chore: better error reporting when connecting to tls with plain socket by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2740 - chore: Support json paths without root selector by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2747 - chore: journal cleanup by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2749 - refactor: remove start-slot-migration cmd [#2727](https://togithub.com/dragonflydb/dragonfly/issues/2727) by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#2728 - feat(server): Add TLS usage to /metrics and `INFO MEMORY` by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2755 - chore: preparations for adding flat json support by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2752 - chore(transaction): Introduce RunCallback by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2760 - feat(replication): Do not auto replicate different master by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2753 - Improve Helm chart to be rendered locally and on machines where is not the application target by [@fafg](https://togithub.com/fafg) in [dragonflydb/dragonfly#2706 - chore: preparation for basic http api by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2764 - feat(server): Add metric for RDB save duration. by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2768 - chore: fix flat_dfs read tests. by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2772 - fix(ci): do not overwrite last_log_file among tests by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2759 - feat(server): support cluster replication by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#2748 - fix: fiber preempts on read path and OnCbFinish() clears fetched_items\_ by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2763 - chore(ci): open last_log_file in append mode by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2776 - doc(README): fix outdated expiry ranges description by [@enjoy-binbin](https://togithub.com/enjoy-binbin) in [dragonflydb/dragonfly#2779 - Benchmark runner by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#2780 - chore(replication-tests): add cache_mode on test replication all by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2685 - feat(tiering): DiskStorage by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2770 - chore: add a boilerplate for bloom filter family by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2782 - chore: introduce conversion routines between JsonType and FlatJson by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2785 - chore: Fix memcached flags not updated by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2787 - chore: remove duplicate code from dash and simplify by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2765 - chore: disable test_cluster_slot_migration by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2788 - fix: new\[] delete\[] missmatch in disk_storage by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2792 - fix: sanitizers clang build and clean up some warnings by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2793 - chore: add bloom filter class by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2791 - chore: add SBF data structure by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2795 - chore(tiering): Disable compilation for MacOs by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#2799 - chore: fix daily build by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2798 - chore: expose SBF via compact_object by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2797 - fix(ci): malloc trim on sanitizers workflow by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2794 - fix(cluster): Don't miss updates in FLUSHSLOTS by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#2783 - feat: add bf.(m)add and bf.(m)exists commands by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2801 - fix: SBF memory leaks by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2803 - chore: refactor StringFamily::Set to use CmdArgParser by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2800 - fix: propagate memcached flags to replica by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2807 - DFLYMIGRATE ACK refactoring by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#2790 - feat: add master lsn and journal_executed dcheck in replica via ping by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2778 - fix: correct json response for errors by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2813 - chore: bloom test - cover corner cases by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2806 - bug(server): do not write lsn opcode to journal by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#2814 - chore: Fix build by disabling the tests. by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2821 - fix(replication): replication with multi shard sync enabled lagging by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#2823 - fix: io_uring/fibers bug in DnsResolve by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#2825 ##### New Contributors - [@manojks1999](https://togithub.com/manojks1999) made their first contribution in [dragonflydb/dragonfly#2659 - [@fafg](https://togithub.com/fafg) made their first contribution in [dragonflydb/dragonfly#2706 - [@enjoy-binbin](https://togithub.com/enjoy-binbin) made their first contribution in [dragonflydb/dragonfly#2779 ##### Huge thanks to all the contributors! ❤️ 🇮🇱 🇺🇦 **Full Changelog**: dragonflydb/dragonfly@v1.15.0...v1.16.0 </details>  Co-authored-by: repo-jeeves[bot] <106431701+repo-jeeves[bot]@users.noreply.github.com>

chore: make usan asan optional and enable them on CI

1f232dd

kostasrim self-assigned this Feb 20, 2024

kostasrim added 20 commits February 26, 2024 18:58

Merge branch 'main' into enable_asan_usan_on_ci

cb10d7e

remove ub

f44dc3a

revert last commit and disable sanitizers on ubuntu-dev-20 container

6715f82

Merge branch 'main' into enable_asan_usan_on_ci

df8c76f

add daily job for sanitizers

4c440e0

test

1a4d8fd

add missing install dep on ubuntu-dev 22

93ae016

try

a4592aa

remove apt install

ba5e2b2

add disk space info

40fcbd1

add cleanup

430654f

run sanitizers on self hosted runner

21f23ab

fix broken value initilization on arm

ef696a1

print ulimit on docker

09a8560

fix failure

30be155

speed up ci

b87f6a6

comment out ci completely

d653d3e

remove search_family from running

d7609ec

revert and clean up

ac26721

remove multi_test from sanitizers and revert temporary fix

ee79e22

kostasrim requested a review from adiholden February 29, 2024 08:18

fix regex string

e24197e

kostasrim requested a review from dranikpg February 29, 2024 09:07

adiholden reviewed Feb 29, 2024

View reviewed changes

kostasrim added 5 commits February 29, 2024 15:10

address comments

f117b0e

remove semicolon from CMakeLists file

208f430

remove more semicolons

9385625

fix Cmake

da8b44b

failing test

99b1604

adiholden reviewed Feb 29, 2024

View reviewed changes

kostasrim added 4 commits February 29, 2024 16:43

test all

96657bf

address gh comments and try gh runner

9f44e2b

add j4 on ninja

f45f246

rever runner

3e9b620

kostasrim requested a review from adiholden February 29, 2024 15:43

kostasrim added 2 commits February 29, 2024 17:43

revert -j4

d8d27b1

Merge branch 'main' into enable_asan_usan_on_ci

a72fd96

adiholden approved these changes Mar 4, 2024

View reviewed changes

kostasrim merged commit f3ba448 into main Mar 4, 2024
10 checks passed

kostasrim deleted the enable_asan_usan_on_ci branch March 4, 2024 09:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: make usan asan optional and enable them on CI #2631

chore: make usan asan optional and enable them on CI #2631

kostasrim commented Feb 20, 2024 •

edited

kostasrim commented Feb 20, 2024

adiholden Feb 29, 2024

adiholden Feb 29, 2024

kostasrim Feb 29, 2024

adiholden Feb 29, 2024

kostasrim Feb 29, 2024

adiholden Feb 29, 2024

kostasrim Feb 29, 2024

adiholden Feb 29, 2024

kostasrim Feb 29, 2024 •

edited

kostasrim Feb 29, 2024

adiholden Feb 29, 2024

kostasrim Feb 29, 2024 •

edited

romange Feb 29, 2024

adiholden Feb 29, 2024

adiholden Feb 29, 2024

kostasrim Feb 29, 2024

kostasrim Feb 29, 2024

adiholden Feb 29, 2024

adiholden Feb 29, 2024

kostasrim Feb 29, 2024

chore: make usan asan optional and enable them on CI #2631

chore: make usan asan optional and enable them on CI #2631

Conversation

kostasrim commented Feb 20, 2024 • edited

kostasrim commented Feb 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kostasrim Feb 29, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kostasrim Feb 29, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kostasrim commented Feb 20, 2024 •

edited

kostasrim Feb 29, 2024 •

edited

kostasrim Feb 29, 2024 •

edited