Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

append log for upgrade and sqlExecutoer #16575

Merged

Conversation

qingxinhome
Copy link
Contributor

@qingxinhome qingxinhome commented Jun 1, 2024

User description

What type of PR is this?

  • API-change
  • BUG
  • Improvement
  • Documentation
  • Feature
  • Test and CI
  • Code Refactoring

Which issue(s) this PR fixes:

issue https://github.com/matrixorigin/MO-Cloud/issues/3404

What this PR does / why we need it:

append log for upgrader and sqlExecutor


PR Type

Bug fix, Enhancement


Description

  • Added error and info logging for pre-sql, upgrade sql, and post-sql execution in upgrade_strategy.go.
  • Added timing and info logging for tenant and cluster upgrade entries in v1_2_1/upgrade.go.
  • Added logging for received SQL execute requests in sql_executor.go.

Changes walkthrough 📝

Relevant files
Enhancement
upgrade_strategy.go
Add logging for SQL execution in upgrade strategy               

pkg/bootstrap/versions/upgrade_strategy.go

  • Added error logging for pre-sql, upgrade sql, and post-sql execution.
  • Added info logging for successful execution of pre-sql, upgrade sql,
    and post-sql.
  • +5/-0     
    upgrade.go
    Add timing and logging for tenant and cluster upgrades     

    pkg/bootstrap/versions/v1_2_1/upgrade.go

  • Added timing for tenant and cluster upgrade entries.
  • Added info logging for completion of tenant and cluster upgrade
    entries.
  • +18/-0   
    sql_executor.go
    Add logging for received SQL execute requests                       

    pkg/sql/compile/sql_executor.go

    • Added logging for received SQL execute requests.
    +4/-0     

    💡 PR-Agent usage:
    Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

    Copy link

    PR-Agent was enabled for this repository. To continue using it, please link your git user with your CodiumAI identity here.

    PR Review 🔍

    ⏱️ Estimated effort to review [1-5]

    2, because the changes are straightforward and mainly involve adding logging statements to existing functions. The logic of the code remains largely unchanged, making it easier to review.

    🧪 Relevant tests

    No

    ⚡ Possible issues

    Possible Performance Issue: The addition of logging, especially with detailed information like SQL statements and transaction IDs, could potentially impact performance, especially if these logs are generated at a high frequency in a production environment.

    🔒 Security concerns

    No

    @matrix-meow matrix-meow added the size/S Denotes a PR that changes [10,99] lines label Jun 1, 2024
    @mergify mergify bot added the kind/bug Something isn't working label Jun 1, 2024
    Copy link

    PR-Agent was enabled for this repository. To continue using it, please link your git user with your CodiumAI identity here.

    PR Code Suggestions ✨

    CategorySuggestion                                                                                                                                    Score
    Best practice
    Use defer to ensure that res.Close() is always called

    To ensure that the res.Close() method is always called, even if an error occurs, it would
    be better to use a defer statement right after the txn.Exec call.

    pkg/bootstrap/versions/upgrade_strategy.go [144-149]

     res, err := txn.Exec(u.PreSql, executor.StatementOption{}.WithAccountID(accountId))
     if err != nil {
         getLogger().Error("execute upgrade entry pre-sql error", zap.Error(err), zap.String("upgrade entry", u.String()))
         return err
     }
    -res.Close()
    +defer res.Close()
     
    Suggestion importance[1-10]: 8

    Why: Using defer for res.Close() right after the txn.Exec call ensures the resource is always properly closed, even if an error occurs, which is a significant improvement in resource management.

    8
    Possible bug
    Add nil checks before accessing nested fields to prevent potential nil pointer dereference

    To avoid potential nil pointer dereference, check if exec.opts.Txn() and
    exec.opts.Txn().Txn() are not nil before calling
    hex.EncodeToString(exec.opts.Txn().Txn().ID).

    pkg/sql/compile/sql_executor.go [259-261]

    +txnID := ""
    +if exec.opts.Txn() != nil && exec.opts.Txn().Txn() != nil {
    +    txnID = hex.EncodeToString(exec.opts.Txn().Txn().ID)
    +}
     logutil.Info("Received SQL execute request",
         zap.String("sql", sql),
    -    zap.String("txn-id", hex.EncodeToString(exec.opts.Txn().Txn().ID)))
    +    zap.String("txn-id", txnID))
     
    Suggestion importance[1-10]: 7

    Why: Adding nil checks before dereferencing pointers is crucial to prevent runtime errors in Go. This suggestion correctly identifies a potential bug and offers a solution to improve code robustness.

    7
    Maintainability
    Extract logging logic into a reusable function to avoid code duplication

    To avoid code duplication and improve maintainability, extract the logging logic into a
    separate function that can be reused.

    pkg/bootstrap/versions/upgrade_strategy.go [146-150]

    -getLogger().Error("execute upgrade entry pre-sql error", zap.Error(err), zap.String("upgrade entry", u.String()))
    -getLogger().Info("execute upgrade pre-sql completed", zap.String("upgrade entry", u.String()))
    +logUpgradeResult := func(err error, stage string) {
    +    if err != nil {
    +        getLogger().Error("execute upgrade entry " + stage + " error", zap.Error(err), zap.String("upgrade entry", u.String()))
    +    } else {
    +        getLogger().Info("execute upgrade " + stage + " completed", zap.String("upgrade entry", u.String()))
    +    }
    +}
    +logUpgradeResult(err, "pre-sql")
     
    Suggestion importance[1-10]: 6

    Why: Extracting repeated logging logic into a separate function enhances maintainability and reduces code duplication. This is a good practice, especially in larger codebases where similar logging patterns are used frequently.

    6
    Enhancement
    Rename the variable duration to elapsedTime for better readability

    To improve readability and avoid potential confusion, rename the variable duration to
    elapsedTime to better reflect its purpose.

    pkg/bootstrap/versions/v1_2_1/upgrade.go [71-76]

    -duration := time.Since(start)
    +elapsedTime := time.Since(start)
     getLogger().Info("tenant upgrade entry complete",
         zap.String("upgrade entry", upgEntry.String()),
    -    zap.Int64("time cost", duration.Milliseconds()),
    +    zap.Int64("time cost", elapsedTime.Milliseconds()),
         zap.Int32("tenantId", tenantID),
         zap.String("toVersion", v.Metadata().Version))
     
    Suggestion importance[1-10]: 4

    Why: While renaming duration to elapsedTime improves readability slightly, it's a minor enhancement and doesn't impact functionality or performance.

    4

    @mergify mergify bot merged commit 0cfeb72 into matrixorigin:main Jun 3, 2024
    17 of 18 checks passed
    XuPeng-SH pushed a commit to XuPeng-SH/matrixone that referenced this pull request Jun 4, 2024
    * GC needs to consume all the mo_snapshot tables (matrixorigin#16539)
    
    Each tenant of the current mo has a mo_snapshot table to store snapshot information. GC needs to consume all mo_snapshot tables.
    
    Approved by: @XuPeng-SH
    
    * append log for upgrade and sqlExecutoer (matrixorigin#16575)
    
    append log for upgrader and sqlExecutor
    
    Approved by: @daviszhen, @badboynt1, @zhangxu19830126, @m-schen
    
    * [enhancement] proxy: filter CNs that are not in working state. (matrixorigin#16558)
    
    1. filter CNs that are not in working state.
    2. add some logs for migration
    
    Approved by: @zhangxu19830126
    
    * fix lock service ut (matrixorigin#16517)
    
    fix lock service ut
    
    Approved by: @zhangxu19830126
    
    * Add cost of GC Check (matrixorigin#16470)
    
    To avoid List() operations on oss, tke or s3, you need to add the Cost interface.
    
    Approved by: @reusee, @XuPeng-SH
    
    * optimize explain info for tp/ap query (matrixorigin#16578)
    
    optimize explain info for tp/ap query
    
    Approved by: @daviszhen, @ouyuanning, @aunjgr
    
    * Bvt disable trace (matrixorigin#16581)
    
    aim to exclude the `system,system_metrics` part case.
    changes:
    1. move `cases/table/system_table_cases` system,system_metrics part into individule case file.
    
    Approved by: @heni02
    
    * remove log print from automaxprocs (matrixorigin#16546)
    
    remove log print from automaxprocs
    
    Approved by: @triump2020, @m-schen, @ouyuanning, @aunjgr, @zhangxu19830126
    
    * rmTag15901 (matrixorigin#16585)
    
    rm 15901
    
    Approved by: @heni02
    
    * remove some MustStrCol&MustBytesCol (matrixorigin#16361)
    
    Remove some unnecessary MustStrCol, MustBytesCol calls.
    
    Approved by: @daviszhen, @reusee, @m-schen, @aunjgr, @XuPeng-SH
    
    * add bvt tag (matrixorigin#16589)
    
    add bvt tag
    
    Approved by: @heni02, @aressu1985
    
    * fix a bug that cause load performance regression issue (matrixorigin#16600)
    
    fix a bug that cause load performance regression issue
    
    Approved by: @m-schen
    
    * add case for restore pub_sub (matrixorigin#16602)
    
    add case for restore pub_sub
    
    Approved by: @heni02
    
    * add shard service kernel (matrixorigin#16565)
    
    Add shardservice kernel.
    
    Approved by: @reusee, @m-schen, @daviszhen, @XuPeng-SH, @volgariver6, @badboynt1, @ouyuanning, @triump2020, @w-zr, @sukki37, @aunjgr, @fengttt
    
    * [BugFix]: Use L2DistanceSq instead of L2Distance during IndexScan (matrixorigin#16366)
    
    During `KNN Select` and `Mapping Entries to Centroids via CROSS_JOIN_L2`, we can make use of L2DistanceSq instead of L2Distance, as it avoids `Sqrt()`. We can see the improvement in QPS for SIFT128 from 90 to 100. However, for GIST960, the QPS did not change much.
    
    L2DistanceSq is suitable only when there is a comparison (ie ORDER BY), and when the absolute value (ie actual L2Distance) is not required.
    - In the case of `CROSS JOIN L2` we find the nearest centroid for the Entry using `L2DistanceSq`. `CROSS JOIN L2` is used in both INSERT and CREATE INDEX.
    - In the case of `KNN SELECT`, our query has ORDER BY L2_DISTANCE(...), which can make use of `L2DistanceSq` as the L2Distance value is not explicitly required.
    
    **NOTE:** L2DistanceSq is not suitable in Kmenas++ for Centroid Computation, as it will impact the centroids picked.
    
    Approved by: @heni02, @m-schen, @aunjgr, @badboynt1
    
    * add sharding metrics (matrixorigin#16606)
    
    add sharding metrics
    
    Approved by: @aptend
    
    * fix data race (matrixorigin#16608)
    
    fix data race
    
    Approved by: @reusee
    
    * Refactor reshape (matrixorigin#15879)
    
    Reshape objects block by block.
    
    Approved by: @XuPeng-SH
    
    * refactor system variables to support account isolation (matrixorigin#16551)
    
    - system variable now is account isolated
    - table `mo_mysql_compatibility_mode` only saves delta info between account's and cluster's default system variable values
    - always use session variable except `show global variables`
    
    Approved by: @daviszhen, @aunjgr, @aressu1985
    
    * fix merge
    
    * [cherry-pick-16594] : fix moc3399 (matrixorigin#16611)
    
    When truncate table, if the table does not have any auto-incr col, there is no need to call the Reset interface of increment_service
    
    Approved by: @ouyuanning
    
    * bump go to 1.22.3, fix make compose and optimize ut script (matrixorigin#16604)
    
    1. bump go version from 1.21.5 to 1.22.3
    2. fix `make compose` to make it work
    3. `make ut` will read `UT_WORKDIR` env variable to store report, it will be `$HOME` if `UT_WORKDIR` is empty
    
    Approved by: @zhangxu19830126, @sukki37
    
    * remove isMerge from build operator (matrixorigin#16622)
    
    remove isMerge from build operator
    
    Approved by: @m-schen
    
    ---------
    
    Co-authored-by: GreatRiver <2552853833@qq.com>
    Co-authored-by: qingxinhome <70939751+qingxinhome@users.noreply.github.com>
    Co-authored-by: LiuBo <g.user.lb@gmail.com>
    Co-authored-by: iamlinjunhong <49111204+iamlinjunhong@users.noreply.github.com>
    Co-authored-by: nitao <badboynt@126.com>
    Co-authored-by: Jackson <xzxiong@yeah.net>
    Co-authored-by: Ariznawlll <ariznawl@163.com>
    Co-authored-by: Wei Ziran <weiziran125@gmail.com>
    Co-authored-by: YANGGMM <www.yangzhao123@gmail.com>
    Co-authored-by: fagongzi <zhangxu19830126@gmail.com>
    Co-authored-by: Arjun Sunil Kumar <arjunsk@users.noreply.github.com>
    Co-authored-by: Kai Cao <ck89119@users.noreply.github.com>
    Co-authored-by: Jensen <jensenojs@qq.com>
    Co-authored-by: brown <endeavorjia@gmail.com>
    @qingxinhome qingxinhome deleted the ClusterUpgradeViewLog_infoHang branch June 12, 2024 01:50
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    Bug fix Enhancement kind/bug Something isn't working Review effort [1-5]: 2 size/S Denotes a PR that changes [10,99] lines
    Projects
    None yet
    Development

    Successfully merging this pull request may close these issues.

    None yet

    6 participants