Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bump go to 1.22.3, fix make compose and optimize ut script #16604

Merged
merged 5 commits into from
Jun 4, 2024

Conversation

guguducken
Copy link
Collaborator

@guguducken guguducken commented Jun 3, 2024

User description

What type of PR is this?

  • API-change
  • BUG
  • Improvement
  • Documentation
  • Feature
  • Test and CI
  • Code Refactoring

Which issue(s) this PR fixes:

issue # https://github.com/matrixorigin/MO-Cloud/issues/3413 https://github.com/matrixorigin/MO-Cloud/issues/3397

What this PR does / why we need it:

  1. bump go version from 1.21.5 to 1.22.3
  2. fix make compose to make it work
  3. make ut will read UT_WORKDIR env variable to store report, it will be $HOME if UT_WORKDIR is empty

PR Type

Enhancement, Tests


Description

  • Bumped Go version from 1.21.5 to 1.22.3 across multiple Dockerfiles and go.mod.
  • Updated log export and Docker image handling in compose_bvt.sh.
  • Improved database readiness check and updated test resource paths in entrypoint.sh.
  • Changed default COMPOSE_LAUNCH value and added cleanup commands in Makefile.
  • Removed version declaration from Docker Compose file.

Changes walkthrough 📝

Relevant files
Enhancement
compose_bvt.sh
Update log export and Docker image handling in compose_bvt.sh

optools/compose_bvt/compose_bvt.sh

  • Removed .bak file deletion in export_logs function.
  • Updated file extensions for stack dump logs.
  • Changed Docker image tag format in compose_bvt function.
  • Removed redundant exit $? statement.
  • +5/-8     
    entrypoint.sh
    Improve database readiness check and update test resource paths

    optools/compose_bvt/entrypoint.sh

  • Modified database readiness check logic in run_bvt function.
  • Updated paths for test resources in run_bvt function.
  • +5/-3     
    Makefile
    Update compose launch configuration and cleanup commands 

    Makefile

  • Changed default COMPOSE_LAUNCH value.
  • Added forced removal of Docker volume in compose-clean.
  • Added cleanup commands for test resources in compose-clean.
  • +5/-2     
    Configuration changes
    compose.yaml
    Remove version declaration from Docker Compose file           

    etc/launch-tae-compose/compose.yaml

    • Removed version declaration from compose.yaml.
    +0/-2     
    Dependencies
    go.mod
    Bump Go version to 1.22.3                                                               

    go.mod

    • Bumped Go version from 1.21.5 to 1.22.3.
    +1/-1     
    Dockerfile
    Update base image to Go 1.22.3 in bvt_ut Dockerfile           

    optools/bvt_ut/Dockerfile

    • Updated base image to use Go 1.22.3.
    +1/-1     
    Dockerfile.tester
    Update base image to Go 1.22.3 in Dockerfile.tester           

    optools/compose_bvt/Dockerfile.tester

  • Updated base image to use Go 1.22.3.
  • Removed installation of mariadb-client.
  • +2/-3     
    Dockerfile
    Update base image to Go 1.22.3 in images Dockerfile           

    optools/images/Dockerfile

    • Updated base image to use Go 1.22.3.
    +1/-1     

    💡 PR-Agent usage:
    Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

    Copy link

    PR-Agent was enabled for this repository. To continue using it, please link your git user with your CodiumAI identity here.

    PR Review 🔍

    ⏱️ Estimated effort to review [1-5]

    3, because the PR involves multiple files and changes across scripts, Dockerfiles, and configuration files. The changes are moderate in complexity, involving updates to version numbers, script logic, and Docker configurations. Reviewing this PR requires understanding the impact of these changes on the build and deployment process, as well as ensuring that the updated scripts and Docker configurations function as expected.

    🧪 Relevant tests

    No

    ⚡ Possible issues

    Possible Bug: The removal of the .bak file deletion in compose_bvt.sh might lead to accumulation of backup files if they are not managed elsewhere.

    Possible Bug: Changing the database readiness check in entrypoint.sh to only check for the presence of databases instead of specific tables might not be sufficient to ensure the database is fully ready for operations.

    🔒 Security concerns

    No

    Copy link

    PR-Agent was enabled for this repository. To continue using it, please link your git user with your CodiumAI identity here.

    PR Code Suggestions ✨

    CategorySuggestion                                                                                                                                    Score
    Possible issue
    Add error handling for curl commands to log failures when fetching debug profiles

    Add error handling for the curl commands to ensure that any issues with fetching the debug
    profiles are caught and logged.

    optools/compose_bvt/compose_bvt.sh [10-11]

    -curl http://localhost:12345/debug/pprof/goroutine\?debug=2 -o docker-compose-log/cn-0-dump-stacks
    -curl http://localhost:22345/debug/pprof/goroutine\?debug=2 -o docker-compose-log/cn-1-dump-stacks
    +curl http://localhost:12345/debug/pprof/goroutine\?debug=2 -o docker-compose-log/cn-0-dump-stacks || echo "Failed to fetch debug profile from cn-0"
    +curl http://localhost:22345/debug/pprof/goroutine\?debug=2 -o docker-compose-log/cn-1-dump-stacks || echo "Failed to fetch debug profile from cn-1"
     
    Suggestion importance[1-10]: 7

    Why: Adding error handling to the curl commands is a good practice to ensure robustness in case of network or server issues. The suggestion correctly identifies the new code and proposes a useful improvement.

    7
    Performance
    Add a sleep interval inside the while true loop to prevent excessive CPU usage

    Add a sleep interval inside the while true loop to avoid a tight loop that could consume
    excessive CPU resources.

    optools/compose_bvt/entrypoint.sh [19-25]

     while true; do
       if mysql -h cn0 -P 6001 -u dump -p111 --execute "show databases;"; then
         break;
       fi
       if [ $i -ge 300 ]; then
         echo "wait for $i seconds, mo init not finish, so exit 1"
         docker ps
    +  fi
    +  sleep 1
    +  i=$(($i+1))
    +done
     
    Suggestion importance[1-10]: 6

    Why: The suggestion to add a sleep interval is valid and would help in reducing CPU usage. However, the suggested code already includes a sleep command and increment of the counter outside the if block, which makes this suggestion partially redundant.

    6
    Maintainability
    Use a variable to store the common URL part for the curl commands to avoid repetition

    Use a variable to store the common URL part for the curl commands to avoid repetition and
    make future changes easier.

    optools/compose_bvt/compose_bvt.sh [10-11]

    -curl http://localhost:12345/debug/pprof/goroutine\?debug=2 -o docker-compose-log/cn-0-dump-stacks
    -curl http://localhost:22345/debug/pprof/goroutine\?debug=2 -o docker-compose-log/cn-1-dump-stacks
    +local base_url="http://localhost"
    +curl ${base_url}:12345/debug/pprof/goroutine\?debug=2 -o docker-compose-log/cn-0-dump-stacks
    +curl ${base_url}:22345/debug/pprof/goroutine\?debug=2 -o docker-compose-log/cn-1-dump-stacks
     
    Suggestion importance[1-10]: 6

    Why: Using a variable for the common URL part is a good practice for maintainability. The suggestion correctly targets the new code and offers a clear improvement.

    6
    Best practice
    Use $(RM) instead of rm -rf for removing directories and files for better portability

    Use $(RM) instead of rm -rf for removing directories and files to leverage the built-in
    make variable for better portability.

    Makefile [209-211]

    -@cd $(ROOT_DIR) && rm -rf docker-compose-log && rm -rf test/distributed/resources/json/export*
    -@cd $(ROOT_DIR) && rm -rf test/distributed/resources/into_outfile/*.csv
    -@cd $(ROOT_DIR) && rm -rf test/distributed/resources/into_outfile_2/*.csv
    +@cd $(ROOT_DIR) && $(RM) -rf docker-compose-log && $(RM) -rf test/distributed/resources/json/export*
    +@cd $(ROOT_DIR) && $(RM) -rf test/distributed/resources/into_outfile/*.csv
    +@cd $(ROOT_DIR) && $(RM) -rf test/distributed/resources/into_outfile_2/*.csv
     
    Suggestion importance[1-10]: 5

    Why: The suggestion to use $(RM) instead of rm -rf is a good practice for portability in Makefiles. However, it's a minor improvement and not critical for functionality.

    5

    @guguducken guguducken changed the title bump go to 1.22.3 and fix make coompose bump go to 1.22.3 and fix make compose Jun 4, 2024
    @guguducken guguducken changed the title bump go to 1.22.3 and fix make compose bump go to 1.22.3, fix make compose and optimized ut script Jun 4, 2024
    @guguducken guguducken changed the title bump go to 1.22.3, fix make compose and optimized ut script bump go to 1.22.3, fix make compose and optimize ut script Jun 4, 2024
    @mergify mergify bot merged commit 13082c7 into matrixorigin:main Jun 4, 2024
    18 checks passed
    guguducken added a commit to guguducken/matrixone that referenced this pull request Jun 4, 2024
    …gin#16604)
    
    1. bump go version from 1.21.5 to 1.22.3
    2. fix `make compose` to make it work
    3. `make ut` will read `UT_WORKDIR` env variable to store report, it will be `$HOME` if `UT_WORKDIR` is empty
    
    Approved by: @zhangxu19830126, @sukki37
    @guguducken guguducken deleted the bump-go-version branch June 4, 2024 13:50
    XuPeng-SH pushed a commit to XuPeng-SH/matrixone that referenced this pull request Jun 4, 2024
    * GC needs to consume all the mo_snapshot tables (matrixorigin#16539)
    
    Each tenant of the current mo has a mo_snapshot table to store snapshot information. GC needs to consume all mo_snapshot tables.
    
    Approved by: @XuPeng-SH
    
    * append log for upgrade and sqlExecutoer (matrixorigin#16575)
    
    append log for upgrader and sqlExecutor
    
    Approved by: @daviszhen, @badboynt1, @zhangxu19830126, @m-schen
    
    * [enhancement] proxy: filter CNs that are not in working state. (matrixorigin#16558)
    
    1. filter CNs that are not in working state.
    2. add some logs for migration
    
    Approved by: @zhangxu19830126
    
    * fix lock service ut (matrixorigin#16517)
    
    fix lock service ut
    
    Approved by: @zhangxu19830126
    
    * Add cost of GC Check (matrixorigin#16470)
    
    To avoid List() operations on oss, tke or s3, you need to add the Cost interface.
    
    Approved by: @reusee, @XuPeng-SH
    
    * optimize explain info for tp/ap query (matrixorigin#16578)
    
    optimize explain info for tp/ap query
    
    Approved by: @daviszhen, @ouyuanning, @aunjgr
    
    * Bvt disable trace (matrixorigin#16581)
    
    aim to exclude the `system,system_metrics` part case.
    changes:
    1. move `cases/table/system_table_cases` system,system_metrics part into individule case file.
    
    Approved by: @heni02
    
    * remove log print from automaxprocs (matrixorigin#16546)
    
    remove log print from automaxprocs
    
    Approved by: @triump2020, @m-schen, @ouyuanning, @aunjgr, @zhangxu19830126
    
    * rmTag15901 (matrixorigin#16585)
    
    rm 15901
    
    Approved by: @heni02
    
    * remove some MustStrCol&MustBytesCol (matrixorigin#16361)
    
    Remove some unnecessary MustStrCol, MustBytesCol calls.
    
    Approved by: @daviszhen, @reusee, @m-schen, @aunjgr, @XuPeng-SH
    
    * add bvt tag (matrixorigin#16589)
    
    add bvt tag
    
    Approved by: @heni02, @aressu1985
    
    * fix a bug that cause load performance regression issue (matrixorigin#16600)
    
    fix a bug that cause load performance regression issue
    
    Approved by: @m-schen
    
    * add case for restore pub_sub (matrixorigin#16602)
    
    add case for restore pub_sub
    
    Approved by: @heni02
    
    * add shard service kernel (matrixorigin#16565)
    
    Add shardservice kernel.
    
    Approved by: @reusee, @m-schen, @daviszhen, @XuPeng-SH, @volgariver6, @badboynt1, @ouyuanning, @triump2020, @w-zr, @sukki37, @aunjgr, @fengttt
    
    * [BugFix]: Use L2DistanceSq instead of L2Distance during IndexScan (matrixorigin#16366)
    
    During `KNN Select` and `Mapping Entries to Centroids via CROSS_JOIN_L2`, we can make use of L2DistanceSq instead of L2Distance, as it avoids `Sqrt()`. We can see the improvement in QPS for SIFT128 from 90 to 100. However, for GIST960, the QPS did not change much.
    
    L2DistanceSq is suitable only when there is a comparison (ie ORDER BY), and when the absolute value (ie actual L2Distance) is not required.
    - In the case of `CROSS JOIN L2` we find the nearest centroid for the Entry using `L2DistanceSq`. `CROSS JOIN L2` is used in both INSERT and CREATE INDEX.
    - In the case of `KNN SELECT`, our query has ORDER BY L2_DISTANCE(...), which can make use of `L2DistanceSq` as the L2Distance value is not explicitly required.
    
    **NOTE:** L2DistanceSq is not suitable in Kmenas++ for Centroid Computation, as it will impact the centroids picked.
    
    Approved by: @heni02, @m-schen, @aunjgr, @badboynt1
    
    * add sharding metrics (matrixorigin#16606)
    
    add sharding metrics
    
    Approved by: @aptend
    
    * fix data race (matrixorigin#16608)
    
    fix data race
    
    Approved by: @reusee
    
    * Refactor reshape (matrixorigin#15879)
    
    Reshape objects block by block.
    
    Approved by: @XuPeng-SH
    
    * refactor system variables to support account isolation (matrixorigin#16551)
    
    - system variable now is account isolated
    - table `mo_mysql_compatibility_mode` only saves delta info between account's and cluster's default system variable values
    - always use session variable except `show global variables`
    
    Approved by: @daviszhen, @aunjgr, @aressu1985
    
    * fix merge
    
    * [cherry-pick-16594] : fix moc3399 (matrixorigin#16611)
    
    When truncate table, if the table does not have any auto-incr col, there is no need to call the Reset interface of increment_service
    
    Approved by: @ouyuanning
    
    * bump go to 1.22.3, fix make compose and optimize ut script (matrixorigin#16604)
    
    1. bump go version from 1.21.5 to 1.22.3
    2. fix `make compose` to make it work
    3. `make ut` will read `UT_WORKDIR` env variable to store report, it will be `$HOME` if `UT_WORKDIR` is empty
    
    Approved by: @zhangxu19830126, @sukki37
    
    * remove isMerge from build operator (matrixorigin#16622)
    
    remove isMerge from build operator
    
    Approved by: @m-schen
    
    ---------
    
    Co-authored-by: GreatRiver <2552853833@qq.com>
    Co-authored-by: qingxinhome <70939751+qingxinhome@users.noreply.github.com>
    Co-authored-by: LiuBo <g.user.lb@gmail.com>
    Co-authored-by: iamlinjunhong <49111204+iamlinjunhong@users.noreply.github.com>
    Co-authored-by: nitao <badboynt@126.com>
    Co-authored-by: Jackson <xzxiong@yeah.net>
    Co-authored-by: Ariznawlll <ariznawl@163.com>
    Co-authored-by: Wei Ziran <weiziran125@gmail.com>
    Co-authored-by: YANGGMM <www.yangzhao123@gmail.com>
    Co-authored-by: fagongzi <zhangxu19830126@gmail.com>
    Co-authored-by: Arjun Sunil Kumar <arjunsk@users.noreply.github.com>
    Co-authored-by: Kai Cao <ck89119@users.noreply.github.com>
    Co-authored-by: Jensen <jensenojs@qq.com>
    Co-authored-by: brown <endeavorjia@gmail.com>
    guguducken added a commit to guguducken/matrixone that referenced this pull request Jun 4, 2024
    …gin#16604)
    
    1. bump go version from 1.21.5 to 1.22.3
    2. fix `make compose` to make it work
    3. `make ut` will read `UT_WORKDIR` env variable to store report, it will be `$HOME` if `UT_WORKDIR` is empty
    
    Approved by: @zhangxu19830126, @sukki37
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Projects
    None yet
    Development

    Successfully merging this pull request may close these issues.

    None yet

    4 participants