Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: unexpected interface conversion #58285

Closed
cockroach-teamcity opened this issue Dec 27, 2020 · 14 comments
Closed

roachtest: unexpected interface conversion #58285

cockroach-teamcity opened this issue Dec 27, 2020 · 14 comments
Assignees
@yuzefovich
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

(roachtest).sqlsmith/setup=seed/setting=no-mutations failed on master@4e3a9ccdb9cac89ae32d6248bb9b119b5bc250f7:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/sqlsmith/setup=seed/setting=no-mutations/run_1
	cluster.go:1637,context.go:140,cluster.go:1626,test_runner.go:841: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2544831-1609052773-15-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 2: 6974
		1: 9138
		4: 6448
		3: dead
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1850
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
More

Artifacts: /sqlsmith/setup=seed/setting=no-mutations

See this test on roachdash
powered by pkg/cmd/internal/issues
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Dec 27, 2020
@cockroach-teamcity

This comment has been minimized.

Sign in to view
@yuzefovich
Copy link
Member

The first issue is

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0xe5c5e2]

goroutine 9801 [running]:
panic(0x41ee900, 0x7629040)
	/usr/local/go/src/runtime/panic.go:1064 +0x545 fp=0xc002c3cea0 sp=0xc002c3cdd8 pc=0x4e1b65
runtime.panicmem(...)
	/usr/local/go/src/runtime/panic.go:212
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:742 +0x413 fp=0xc002c3ced0 sp=0xc002c3cea0 pc=0x4f88b3
github.com/cockroachdb/cockroach/pkg/util/tracing.SpanFromContext(0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/tracing/context.go:25 +0x22 fp=0xc002c3cf08 sp=0xc002c3ced0 pc=0xe5c5e2
github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog(0x0, 0x0, 0x3, 0x0, 0xc001a5c6f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/trace.go:96 +0x39 fp=0xc002c3cf48 sp=0xc002c3cf08 pc=0xf47679
github.com/cockroachdb/cockroach/pkg/util/log.vEventf(0x0, 0x0, 0xc002c3d000, 0x1, 0x7f0d00000002, 0x498bc93, 0x57, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/trace.go:246 +0x12a fp=0xc002c3d078 sp=0xc002c3cf48 pc=0xf48bca
github.com/cockroachdb/cockroach/pkg/util/log.VEvent(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/trace.go:267
github.com/cockroachdb/cockroach/pkg/sql/execinfra.MisplannedRanges(0x0, 0x0, 0xc001f313e0, 0x1, 0x1, 0x3, 0xc0008ea5a0, 0x34, 0x34, 0xc002c3d490)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/readerbase.go:65 +0x8c fp=0xc002c3d3b0 sp=0xc002c3d078 pc=0x237056c
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*tableReader).generateMeta(0xc00180b800, 0x0, 0x0, 0x7b9088, 0x555e600, 0xc002084690)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/tablereader.go:311 +0x605 fp=0xc002c3d668 sp=0xc002c3d3b0 pc=0x25f4385
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*tableReader).generateTrailingMeta(0xc00180b800, 0x0, 0x0, 0x203000, 0xc000b91b90, 0x54c9b00)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/tablereader.go:172 +0x45 fp=0xc002c3d6c0 sp=0xc002c3d668 pc=0x25f3065
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*tableReader).generateTrailingMeta-fm(0x0, 0x0, 0x0, 0x0, 0x9e28eb)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/tablereader.go:171 +0x3e fp=0xc002c3d700 sp=0xc002c3d6c0 pc=0x260799e
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBase).moveToTrailingMeta(0xc00180b800)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:718 +0x226 fp=0xc002c3d840 sp=0xc002c3d700 pc=0x236ef06
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBase).MoveToDraining(0xc00180b800, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:611 +0x1b8 fp=0xc002c3d948 sp=0xc002c3d840 pc=0x236e838
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBase).ConsumerDone(0xc00180b800)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:905 +0x33 fp=0xc002c3d970 sp=0xc002c3d948 pc=0x23701b3
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBase).MoveToDraining(0xc001b15200, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:608 +0x18d fp=0xc002c3da78 sp=0xc002c3d970 pc=0x236e80d
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).DrainMeta(0xc001b15200, 0x555e600, 0xc002084480, 0x0, 0x0, 0xc001bf46a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:221 +0x4a fp=0xc002c3db48 sp=0xc002c3da78 pc=0x27e40aa
github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Outbox).sendMetadata(0xc00217e8c0, 0x555e600, 0xc002084480, 0x7f0de9ca6530, 0xc001bf4560, 0x54c6b80, 0xc001bf45c0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:288 +0x111 fp=0xc002c3dde0 sp=0xc002c3db48 pc=0x3130b11
github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Outbox).runWithStream(0xc00217e8c0, 0x555e600, 0xc002084480, 0x7f0de9ca6530, 0xc001bf4560, 0xc001bf44f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:328 +0x170 fp=0xc002c3de58 sp=0xc002c3dde0 pc=0x31311f0
github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Outbox).Run(0xc00217e8c0, 0x555e540, 0xc002b00980, 0x54c5540, 0xc000c0d040, 0x710323e400000001, 0x1ec7aaaa65477e61, 0x50d2032e, 0x0, 0xc001bf44f0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:180 +0x2f7 fp=0xc002c3def8 sp=0xc002c3de58 pc=0x31305d7
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreator).setupRemoteOutputStream.func1(0x555e540, 0xc0025c9c40, 0xc001b2db30)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:652 +0x125 fp=0xc002c3df90 sp=0xc002c3def8 pc=0x313a465
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreatorHelper).accumulateAsyncComponent.func1.1(0xc001f314d0, 0x555e540, 0xc0025c9c40, 0xc001b2db30, 0xc0026cbcf8)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:1266 +0x44 fp=0xc002c3dfb8 sp=0xc002c3df90 pc=0x313cd84
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc002c3dfc0 sp=0xc002c3dfb8 pc=0x51af41
created by github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreatorHelper).accumulateAsyncComponent.func1
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:1265 +0x6f

The second issue is a dup of #39433.

@cockroach-teamcity

This comment has been minimized.

Sign in to view
@yuzefovich
Copy link
Member

The third instance is also #39433, adding a skip of such errors in #58249.

@cockroach-teamcity
Copy link
Member Author

(roachtest).sqlsmith/setup=seed/setting=no-mutations failed on master@6d49a323b52966becfe8a2c38a1a8ccdf8ee58a1:

					FROM
						(
							VALUES
								('1973-06-24 03:51:09.00044':::TIMESTAMP, 485830027:::INT8),
								('2011-10-02 03:28:11.000268':::TIMESTAMP, (-1006388295):::INT8),
								(NULL, 633888716:::INT8),
								('2019-11-05 05:47:54.000909':::TIMESTAMP, (-983875156):::INT8),
								(NULL, (-1023686099):::INT8),
								('2026-09-09 19:11:13.00063':::TIMESTAMP, 108905773:::INT8)
						)
							AS tab_13254 (col_32629, col_32630)
					UNION
						SELECT
							*
						FROM
							(
								VALUES
									(
										'1980-02-16 14:01:47.00093':::TIMESTAMP,
										(
											SELECT
												tab_13255._int4 AS col_32631
											FROM
												defaultdb.public.seed@seed__int8__float8__date_idx AS tab_13255
											WHERE
												(NOT tab_13255._bool)
											ORDER BY
												tab_13255._uuid DESC, tab_13255._uuid DESC, tab_13255._date DESC
											LIMIT
												1:::INT8
										)
									),
									('1977-01-03 10:54:20.000303':::TIMESTAMP, (-1346725564):::INT8),
									(
										CASE
										WHEN false THEN NULL
										ELSE '1995-12-10 12:23:22.000854':::TIMESTAMP
										END,
										417348723:::INT8
									),
									('2025-10-23 07:46:51.000873':::TIMESTAMP, (-2049317317):::INT8)
							)
								AS tab_13256 (col_32632, col_32633)
				)
		SELECT
			cte_ref_1491.col_32629 AS col_32634, (-635522205):::INT8 AS col_32635
		FROM
			with_5438 AS cte_ref_1491
		LIMIT
			13:::INT8;
More

Artifacts: /sqlsmith/setup=seed/setting=no-mutations

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity

This comment has been minimized.

Sign in to view
@cockroach-teamcity
Copy link
Member Author

(roachtest).sqlsmith/setup=seed/setting=no-mutations failed on master@cee475331ca3629b503cd2e7c7919b72c98a5ca5:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/sqlsmith/setup=seed/setting=no-mutations/run_1
	sqlsmith.go:198,sqlsmith.go:228,test_runner.go:760: error: pq: internal error: unable to vectorize execution plan: "$2" IN (NULL::public.greeting, b'\x80':::@100052): unsupported comparison operator: "$2" IN (NULL::public.greeting, b'\x80':::@100052): cannot resolve types in DistSQL by name
		stmt:
		SELECT
			tab_5488._bytes AS col_13586,
			3011640294.402935244:::DECIMAL AS col_13587,
			NULL AS col_13588,
			tab_5488._enum AS col_13589
		FROM
			defaultdb.public.seed@seed__int8__float8__date_idx AS tab_5488
		WHERE
			tab_5488._enum::greeting IN (NULL, 'hi':::greeting)
		ORDER BY
			tab_5488._float4 ASC
		LIMIT
			57:::INT8;
More

Artifacts: /sqlsmith/setup=seed/setting=no-mutations

See this test on roachdash
powered by pkg/cmd/internal/issues

@asubiotto
Copy link
Contributor

The penultimate error is:

pq: internal error: unexpected error from the vectorized engine: interface conversion: coldata.Column is coldata.Int64s, not coldata.Int16s

It seems like there are a bunch of errors in this thread. I think we should focus on fixing the interface conversion since the latest seems expected.

@asubiotto asubiotto changed the title roachtest: sqlsmith/setup=seed/setting=no-mutations failed roachtest: unexpected interface conversion Jan 11, 2021
@asubiotto asubiotto mentioned this issue Jan 11, 2021
Closed
@yuzefovich yuzefovich mentioned this issue Jan 12, 2021
Closed
@yuzefovich
Copy link
Member

The first occurrence of the conversion issue is reproducible:

CREATE TABLE t (_int4 INT4);
INSERT INTO t VALUES (2);
WITH
  cte (col)
    AS (
      SELECT * FROM (VALUES (1:::INT8))
      UNION
        SELECT
          *
        FROM
          (VALUES ((SELECT _int4 FROM t LIMIT 1)))
    )
SELECT
  col
FROM
  cte;

I'll look into this, and I also filed #58889 with the enum type resolution issue.

@yuzefovich
Copy link
Member

Looking at EXPLAIN (TYPES):

  • root
  │ columns: (col int)
  │
  ├── • render
  │   │ columns: (col int)
  │   │ estimated row count: 2
  │   │ render 0: (column1)[int]
  │   │
  │   └── • union
  │       │ columns: (column1 int)
  │       │ estimated row count: 2
  │       │
  │       ├── • values
  │       │     columns: (column1 int)
  │       │     size: 1 column, 1 row
  │       │     row 0, expr 0: (1)[int]
  │       │
  │       └── • values
  │             columns: (column1 int4)
  │             size: 1 column, 1 row
  │             row 0, expr 0: (@S1)[int4]
  │
  └── • subquery
      │ id: @S1
      │ original sql: (SELECT _int4 FROM t LIMIT 1)
      │ exec mode: one row
      │
      └── • scan
            columns: (_int4 int4)
            estimated row count: 1 (missing stats)
            table: t@primary
            spans: LIMITED SCAN
            limit: 1

we see that we're currently UNIONing INT (treated as INT8) with INT4 which trips up the vectorized engine. We definitely could plan a cast in such scenario during the physical planning (in createPlanForSetOp), but I wonder whether that cast needs to be added by the optimizer. cc @mgartner

@yuzefovich yuzefovich mentioned this issue Jan 15, 2021
Closed
@asubiotto
Copy link
Contributor

What's the status on this? Can we remove the release blocker label for the alpha?

@yuzefovich
Copy link
Member

yuzefovich commented Jan 19, 2021 •
edited
Loading

It's definitely not a release blocker, so I'll remove the label. I also think that the fix should be done at the optimizer level, so I'll put it into their project. cc @RaduBerinde

@yuzefovich yuzefovich removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Jan 19, 2021
@RaduBerinde RaduBerinde mentioned this issue Jan 19, 2021
Closed
@RaduBerinde
Copy link
Member

RaduBerinde commented Jan 19, 2021 •
edited
Loading

I filed a separate issue #59148 since there are other failures in this one.

@yuzefovich yuzefovich mentioned this issue Jan 19, 2021
Closed
@yuzefovich
Copy link
Member

I think all of the issues here are addressed (except for the nil ctx which has a separate issue), so I'll close this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yuzefovich yuzefovich

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@yuzefovich @asubiotto @RaduBerinde @cockroach-teamcity