Skip to content

Commit

Permalink
feat: add inspect report subcommand and embed queries as SQL scripts (
Browse files Browse the repository at this point in the history
#2246)

* initial work to split out queries into standalone scripts

* add report command to export all command output to CSV files and embed
SQL scripts

* add progress printing to report command

* add output flag to specify directory

* Update internal/inspect/utils.go

Co-authored-by: Han Qiao <sweatybridge@gmail.com>

* use pgx for csv output and rename flag for output path

* chore(deps): bump tar from 7.0.1 to 7.1.0 (#2240)

Bumps [tar](https://github.com/isaacs/node-tar) from 7.0.1 to 7.1.0.
- [Release notes](https://github.com/isaacs/node-tar/releases)
- [Changelog](https://github.com/isaacs/node-tar/blob/main/CHANGELOG.md)
- [Commits](isaacs/node-tar@v7.0.1...v7.1.0)

---
updated-dependencies:
- dependency-name: tar
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/golangci/golangci-lint from 1.57.2 to 1.58.0 (#2241)

chore(deps): bump github.com/golangci/golangci-lint

Bumps [github.com/golangci/golangci-lint](https://github.com/golangci/golangci-lint) from 1.57.2 to 1.58.0.
- [Release notes](https://github.com/golangci/golangci-lint/releases)
- [Changelog](https://github.com/golangci/golangci-lint/blob/master/CHANGELOG.md)
- [Commits](golangci/golangci-lint@v1.57.2...v1.58.0)

---
updated-dependencies:
- dependency-name: github.com/golangci/golangci-lint
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump golang.org/x/term from 0.19.0 to 0.20.0 (#2243)

Bumps [golang.org/x/term](https://github.com/golang/term) from 0.19.0 to 0.20.0.
- [Commits](golang/term@v0.19.0...v0.20.0)

---
updated-dependencies:
- dependency-name: golang.org/x/term
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump golang.org/x/oauth2 from 0.19.0 to 0.20.0 (#2242)

Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.19.0 to 0.20.0.
- [Commits](golang/oauth2@v0.19.0...v0.20.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: bump edge-runtime to 1.47.0

* fix: bump studio version (#2245)

* chore(deps): bump github.com/gin-gonic/gin from 1.9.1 to 1.10.0 (#2249)

Bumps [github.com/gin-gonic/gin](https://github.com/gin-gonic/gin) from 1.9.1 to 1.10.0.
- [Release notes](https://github.com/gin-gonic/gin/releases)
- [Changelog](https://github.com/gin-gonic/gin/blob/master/CHANGELOG.md)
- [Commits](gin-gonic/gin@v1.9.1...v1.10.0)

---
updated-dependencies:
- dependency-name: github.com/gin-gonic/gin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: bump edge-runtime to 1.48.0

* chore(deps): bump golangci/golangci-lint-action from 5 to 6 (#2248)

Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 5 to 6.
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](golangci/golangci-lint-action@v5...v6)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: clean up db pull implementation (#2175)

* chore: clean up db pull implementation

* chore: update unit tests

* feat: add hook secrets

* Apply suggestions from code review

* fix: add send sms hook as test

* chore: typo in variable name

* fix: index out of bound when applying migrations

* chore(deps): bump github.com/golangci/golangci-lint from 1.58.0 to 1.58.1 (#2256)

chore(deps): bump github.com/golangci/golangci-lint

Bumps [github.com/golangci/golangci-lint](https://github.com/golangci/golangci-lint) from 1.58.0 to 1.58.1.
- [Release notes](https://github.com/golangci/golangci-lint/releases)
- [Changelog](https://github.com/golangci/golangci-lint/blob/master/CHANGELOG.md)
- [Commits](golangci/golangci-lint@v1.58.0...v1.58.1)

---
updated-dependencies:
- dependency-name: github.com/golangci/golangci-lint
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/charmbracelet/bubbletea from 0.26.1 to 0.26.2 (#2257)

chore(deps): bump github.com/charmbracelet/bubbletea

Bumps [github.com/charmbracelet/bubbletea](https://github.com/charmbracelet/bubbletea) from 0.26.1 to 0.26.2.
- [Release notes](https://github.com/charmbracelet/bubbletea/releases)
- [Commits](charmbracelet/bubbletea@v0.26.1...v0.26.2)

---
updated-dependencies:
- dependency-name: github.com/charmbracelet/bubbletea
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: support struct return type for pgmock

* fix: add unit test for cache hit

* fix: create migrations dir before saving diff (#2260)

* chore(deps): bump github.com/docker/cli from 26.1.1+incompatible to 26.1.2+incompatible (#2261)

chore(deps): bump github.com/docker/cli

Bumps [github.com/docker/cli](https://github.com/docker/cli) from 26.1.1+incompatible to 26.1.2+incompatible.
- [Commits](docker/cli@v26.1.1...v26.1.2)

---
updated-dependencies:
- dependency-name: github.com/docker/cli
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/containers/common from 0.58.2 to 0.58.3 (#2262)

Bumps [github.com/containers/common](https://github.com/containers/common) from 0.58.2 to 0.58.3.
- [Release notes](https://github.com/containers/common/releases)
- [Commits](containers/common@v0.58.2...v0.58.3)

---
updated-dependencies:
- dependency-name: github.com/containers/common
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/docker/docker from 26.1.1+incompatible to 26.1.2+incompatible (#2263)

chore(deps): bump github.com/docker/docker

Bumps [github.com/docker/docker](https://github.com/docker/docker) from 26.1.1+incompatible to 26.1.2+incompatible.
- [Release notes](https://github.com/docker/docker/releases)
- [Commits](moby/moby@v26.1.1...v26.1.2)

---
updated-dependencies:
- dependency-name: github.com/docker/docker
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: bump edge-runtime to 1.49.0

* initial work to split out queries into standalone scripts

* add test cases for inspect commands

* chore: use pg schemas in unit tests

* chore: test report command

* chore: verify connection is closed

* chore: embed queries within each command package

* chore: move query reading to file walker

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Han Qiao <sweatybridge@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Lakshan Perera <lakshan@laktek.com>
Co-authored-by: Terry Sutton <saltcod@gmail.com>
Co-authored-by: joel <lee.yi.jie.joel@gmail.com>
Co-authored-by: Qiao Han <qiao@supabase.io>
Co-authored-by: Nyannyacha <meow@nnc.gg>
  • Loading branch information
8 people committed May 21, 2024
1 parent 91cc4aa commit ee621c5
Show file tree
Hide file tree
Showing 61 changed files with 1,354 additions and 349 deletions.
29 changes: 26 additions & 3 deletions cmd/inspect.go
Original file line number Diff line number Diff line change
@@ -1,16 +1,20 @@
package cmd

import (
"fmt"
"os"
"os/signal"
"path/filepath"

"github.com/spf13/afero"
"github.com/spf13/cobra"
"github.com/supabase/cli/internal/inspect/bloat"
"github.com/supabase/cli/internal/inspect/blocking"
"github.com/supabase/cli/internal/inspect/cache"
"github.com/supabase/cli/internal/utils"
"github.com/supabase/cli/internal/utils/flags"

"github.com/supabase/cli/internal/inspect"
"github.com/supabase/cli/internal/inspect/calls"
"github.com/supabase/cli/internal/inspect/index_sizes"
"github.com/supabase/cli/internal/inspect/index_usage"
Expand Down Expand Up @@ -197,15 +201,31 @@ var (
return role_connections.Run(cmd.Context(), flags.DbConfig, afero.NewOsFs())
},
}

outputDir string

reportCmd = &cobra.Command{
Use: "report",
Short: "Generate a CSV output for all inspect commands",
RunE: func(cmd *cobra.Command, args []string) error {
if len(outputDir) == 0 {
defaultPath := filepath.Join(utils.CurrentDirAbs, "report")
title := fmt.Sprintf("Enter a directory to save output files (or leave blank to use %s): ", utils.Bold(defaultPath))
if outputDir = utils.NewConsole().PromptText(title); len(outputDir) == 0 {
outputDir = defaultPath
}
}
return inspect.Report(cmd.Context(), outputDir, flags.DbConfig, afero.NewOsFs())
},
}
)

func init() {
inspectFlags := inspectDBCmd.PersistentFlags()
inspectFlags := inspectCmd.PersistentFlags()
inspectFlags.String("db-url", "", "Inspect the database specified by the connection string (must be percent-encoded).")
inspectFlags.Bool("linked", true, "Inspect the linked project.")
inspectFlags.Bool("local", false, "Inspect the local database.")
inspectDBCmd.MarkFlagsMutuallyExclusive("db-url", "linked", "local")
inspectCmd.AddCommand(inspectDBCmd)
inspectCmd.MarkFlagsMutuallyExclusive("db-url", "linked", "local")
inspectDBCmd.AddCommand(inspectCacheHitCmd)
inspectDBCmd.AddCommand(inspectReplicationSlotsCmd)
inspectDBCmd.AddCommand(inspectIndexUsageCmd)
Expand All @@ -225,5 +245,8 @@ func init() {
inspectDBCmd.AddCommand(inspectBloatCmd)
inspectDBCmd.AddCommand(inspectVacuumStatsCmd)
inspectDBCmd.AddCommand(inspectRoleConnectionsCmd)
inspectCmd.AddCommand(inspectDBCmd)
reportCmd.Flags().StringVar(&outputDir, "output-dir", "", "Path to save CSV files in")
inspectCmd.AddCommand(reportCmd)
rootCmd.AddCommand(inspectCmd)
}
8 changes: 6 additions & 2 deletions internal/inspect/bloat/bloat.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,22 @@ package bloat

import (
"context"
_ "embed"
"fmt"

"github.com/go-errors/errors"
"github.com/jackc/pgconn"
"github.com/jackc/pgx/v4"
"github.com/spf13/afero"
"github.com/supabase/cli/internal/db/reset"
"github.com/supabase/cli/internal/inspect"
"github.com/supabase/cli/internal/migration/list"
"github.com/supabase/cli/internal/utils"
"github.com/supabase/cli/internal/utils/pgxv5"
)

//go:embed bloat.sql
var BloatQuery string

type Result struct {
Type string
Schemaname string
Expand All @@ -28,7 +31,8 @@ func Run(ctx context.Context, config pgconn.Config, fsys afero.Fs, options ...fu
if err != nil {
return err
}
rows, err := conn.Query(ctx, inspect.BLOAT_QUERY, reset.LikeEscapeSchema(utils.InternalSchemas))
defer conn.Close(context.Background())
rows, err := conn.Query(ctx, BloatQuery, reset.LikeEscapeSchema(utils.InternalSchemas))
if err != nil {
return errors.Errorf("failed to query rows: %w", err)
}
Expand Down
61 changes: 61 additions & 0 deletions internal/inspect/bloat/bloat.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
WITH constants AS (
SELECT current_setting('block_size')::numeric AS bs, 23 AS hdr, 4 AS ma
), bloat_info AS (
SELECT
ma,bs,schemaname,tablename,
(datawidth+(hdr+ma-(case when hdr%ma=0 THEN ma ELSE hdr%ma END)))::numeric AS datahdr,
(maxfracsum*(nullhdr+ma-(case when nullhdr%ma=0 THEN ma ELSE nullhdr%ma END))) AS nullhdr2
FROM (
SELECT
schemaname, tablename, hdr, ma, bs,
SUM((1-null_frac)*avg_width) AS datawidth,
MAX(null_frac) AS maxfracsum,
hdr+(
SELECT 1+count(*)/8
FROM pg_stats s2
WHERE null_frac<>0 AND s2.schemaname = s.schemaname AND s2.tablename = s.tablename
) AS nullhdr
FROM pg_stats s, constants
GROUP BY 1,2,3,4,5
) AS foo
), table_bloat AS (
SELECT
schemaname, tablename, cc.relpages, bs,
CEIL((cc.reltuples*((datahdr+ma-
(CASE WHEN datahdr%ma=0 THEN ma ELSE datahdr%ma END))+nullhdr2+4))/(bs-20::float)) AS otta
FROM bloat_info
JOIN pg_class cc ON cc.relname = bloat_info.tablename
JOIN pg_namespace nn ON cc.relnamespace = nn.oid AND nn.nspname = bloat_info.schemaname AND nn.nspname <> 'information_schema'
), index_bloat AS (
SELECT
schemaname, tablename, bs,
COALESCE(c2.relname,'?') AS iname, COALESCE(c2.reltuples,0) AS ituples, COALESCE(c2.relpages,0) AS ipages,
COALESCE(CEIL((c2.reltuples*(datahdr-12))/(bs-20::float)),0) AS iotta -- very rough approximation, assumes all cols
FROM bloat_info
JOIN pg_class cc ON cc.relname = bloat_info.tablename
JOIN pg_namespace nn ON cc.relnamespace = nn.oid AND nn.nspname = bloat_info.schemaname AND nn.nspname <> 'information_schema'
JOIN pg_index i ON indrelid = cc.oid
JOIN pg_class c2 ON c2.oid = i.indexrelid
)
SELECT
type, schemaname, object_name, bloat, pg_size_pretty(raw_waste) as waste
FROM
(SELECT
'table' as type,
schemaname,
tablename as object_name,
ROUND(CASE WHEN otta=0 THEN 0.0 ELSE table_bloat.relpages/otta::numeric END,1) AS bloat,
CASE WHEN relpages < otta THEN '0' ELSE (bs*(table_bloat.relpages-otta)::bigint)::bigint END AS raw_waste
FROM
table_bloat
UNION
SELECT
'index' as type,
schemaname,
tablename || '::' || iname as object_name,
ROUND(CASE WHEN iotta=0 OR ipages=0 THEN 0.0 ELSE ipages/iotta::numeric END,1) AS bloat,
CASE WHEN ipages < iotta THEN '0' ELSE (bs*(ipages-iotta))::bigint END AS raw_waste
FROM
index_bloat) bloat_summary
WHERE NOT schemaname LIKE ANY($1)
ORDER BY raw_waste DESC, bloat DESC
43 changes: 43 additions & 0 deletions internal/inspect/bloat/bloat_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
package bloat

import (
"context"
"testing"

"github.com/jackc/pgconn"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/supabase/cli/internal/db/reset"
"github.com/supabase/cli/internal/testing/pgtest"
"github.com/supabase/cli/internal/utils"
)

var dbConfig = pgconn.Config{
Host: "127.0.0.1",
Port: 5432,
User: "admin",
Password: "password",
Database: "postgres",
}

func TestBloat(t *testing.T) {
t.Run("inspects bloat", func(t *testing.T) {
// Setup in-memory fs
fsys := afero.NewMemMapFs()
// Setup mock postgres
conn := pgtest.NewConn()
defer conn.Close(t)
conn.Query(BloatQuery, reset.LikeEscapeSchema(utils.InternalSchemas)).
Reply("SELECT 1", Result{
Type: "index hit rate",
Schemaname: "public",
Object_name: "table",
Bloat: "0.9",
Waste: "0.1",
})
// Run test
err := Run(context.Background(), dbConfig, fsys, conn.Intercept)
// Check error
assert.NoError(t, err)
})
}
9 changes: 7 additions & 2 deletions internal/inspect/blocking/blocking.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,22 @@ package blocking

import (
"context"
_ "embed"
"fmt"
"regexp"

"github.com/go-errors/errors"
"github.com/jackc/pgconn"
"github.com/jackc/pgx/v4"
"github.com/spf13/afero"
"github.com/supabase/cli/internal/inspect"
"github.com/supabase/cli/internal/migration/list"
"github.com/supabase/cli/internal/utils"
"github.com/supabase/cli/internal/utils/pgxv5"
)

//go:embed blocking.sql
var BlockingQuery string

type Result struct {
Blocked_pid int
Blocking_statement string
Expand All @@ -29,7 +32,9 @@ func Run(ctx context.Context, config pgconn.Config, fsys afero.Fs, options ...fu
if err != nil {
return err
}
rows, err := conn.Query(ctx, inspect.BLOCKING_QUERY)
defer conn.Close(context.Background())
// Ref: https://github.com/heroku/heroku-pg-extras/blob/main/commands/blocking.js#L7
rows, err := conn.Query(ctx, BlockingQuery)
if err != nil {
return errors.Errorf("failed to query rows: %w", err)
}
Expand Down
15 changes: 15 additions & 0 deletions internal/inspect/blocking/blocking.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
SELECT
bl.pid AS blocked_pid,
ka.query AS blocking_statement,
age(now(), ka.query_start)::text AS blocking_duration,
kl.pid AS blocking_pid,
a.query AS blocked_statement,
age(now(), a.query_start)::text AS blocked_duration
FROM pg_catalog.pg_locks bl
JOIN pg_catalog.pg_stat_activity a
ON bl.pid = a.pid
JOIN pg_catalog.pg_locks kl
JOIN pg_catalog.pg_stat_activity ka
ON kl.pid = ka.pid
ON bl.transactionid = kl.transactionid AND bl.pid != kl.pid
WHERE NOT bl.granted
42 changes: 42 additions & 0 deletions internal/inspect/blocking/blocking_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
package blocking

import (
"context"
"testing"

"github.com/jackc/pgconn"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/supabase/cli/internal/testing/pgtest"
)

var dbConfig = pgconn.Config{
Host: "127.0.0.1",
Port: 5432,
User: "admin",
Password: "password",
Database: "postgres",
}

func TestBloatCommand(t *testing.T) {
t.Run("inspects blocking", func(t *testing.T) {
// Setup in-memory fs
fsys := afero.NewMemMapFs()
// Setup mock postgres
conn := pgtest.NewConn()
defer conn.Close(t)
conn.Query(BlockingQuery).
Reply("SELECT 1", Result{
Blocked_pid: 1,
Blocking_statement: "select 1",
Blocking_duration: "2s",
Blocking_pid: 1,
Blocked_statement: "select 1",
Blocked_duration: "2s",
})
// Run test
err := Run(context.Background(), dbConfig, fsys, conn.Intercept)
// Check error
assert.NoError(t, err)
})
}
8 changes: 6 additions & 2 deletions internal/inspect/cache/cache.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,30 +2,34 @@ package cache

import (
"context"
_ "embed"
"fmt"

"github.com/go-errors/errors"
"github.com/jackc/pgconn"
"github.com/jackc/pgx/v4"
"github.com/spf13/afero"
"github.com/supabase/cli/internal/inspect"
"github.com/supabase/cli/internal/migration/list"
"github.com/supabase/cli/internal/utils"
"github.com/supabase/cli/internal/utils/pgxv5"
)

//go:embed cache.sql
var CacheQuery string

type Result struct {
Name string
Ratio float64
}

func Run(ctx context.Context, config pgconn.Config, fsys afero.Fs, options ...func(*pgx.ConnConfig)) error {
// Ref: https://github.com/heroku/heroku-pg-extras/blob/main/commands/cache_hit.js#L7
conn, err := utils.ConnectByConfig(ctx, config, options...)
if err != nil {
return err
}
defer conn.Close(context.Background())
rows, err := conn.Query(ctx, inspect.CACHE_QUERY)
rows, err := conn.Query(ctx, CacheQuery)
if err != nil {
return errors.Errorf("failed to query rows: %w", err)
}
Expand Down
9 changes: 9 additions & 0 deletions internal/inspect/cache/cache.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
SELECT
'index hit rate' AS name,
(sum(idx_blks_hit)) / nullif(sum(idx_blks_hit + idx_blks_read),0) AS ratio
FROM pg_statio_user_indexes
UNION ALL
SELECT
'table hit rate' AS name,
sum(heap_blks_hit) / nullif(sum(heap_blks_hit) + sum(heap_blks_read),0) AS ratio
FROM pg_statio_user_tables
5 changes: 2 additions & 3 deletions internal/inspect/cache/cache_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ import (
"github.com/jackc/pgconn"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/supabase/cli/internal/inspect"
"github.com/supabase/cli/internal/testing/pgtest"
)

Expand All @@ -26,7 +25,7 @@ func TestCacheCommand(t *testing.T) {
// Setup mock postgres
conn := pgtest.NewConn()
defer conn.Close(t)
conn.Query(inspect.CACHE_QUERY).
conn.Query(CacheQuery).
Reply("SELECT 1", Result{
Name: "index hit rate",
Ratio: 0.9,
Expand All @@ -43,7 +42,7 @@ func TestCacheCommand(t *testing.T) {
// Setup mock postgres
conn := pgtest.NewConn()
defer conn.Close(t)
conn.Query(inspect.CACHE_QUERY).
conn.Query(CacheQuery).
Reply("SELECT 1", []interface{}{})
// Run test
err := Run(context.Background(), dbConfig, fsys, conn.Intercept)
Expand Down

0 comments on commit ee621c5

Please sign in to comment.