Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segfault on analyse in multinode setup. #3733

Closed
tanglebones opened this issue Oct 22, 2021 · 7 comments · Fixed by #3745
Closed

Segfault on analyse in multinode setup. #3733

tanglebones opened this issue Oct 22, 2021 · 7 comments · Fixed by #3745
Assignees
@nikkhils
Labels
multinode postgis segfault Segmentation fault

Comments

@tanglebones
Copy link

Relevant system information:

  • OS: docker desktop on windows 10
  • PostgreSQL version: PostgreSQL 13.3 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.2.1_pre1) 10.2.1 20201203, 64-bit
  • TimescaleDB version: 2.3.0
  • Installation method: using Docker

Describe the bug

I was following the instructions https://github.com/binakot/Multi-Node-TimescaleDB/tree/pgconf2021 to try out timescaledb and it is segfaulting.

To Reproduce

docker network create pg_cluster_network
docker-compose up -d
gzip -k -d ./data/*csv.gz
docker exec -i pg_access_node /bin/sh < ./load-init-data.sh
  • via psql/etc to the access node: analyse telemetries;

Expected behavior

No crashing

Actual behavior

Segfault on data node:

2021-10-22 23:06:42.620 UTC [1] LOG:  server process (PID 240) was terminated by signal 11: Segmentation fault
2021-10-22 23:06:42.620 UTC [1] DETAIL:  Failed process was running: SELECT chunk_id,hypertable_id,att_num,nullfrac,width,distinctval,slotkind,slotopstrings,slotcollations,slot1numbers,slot2numbers,slot3numbers,slot4numbers,slot5numbers,slotvaluetypetrings,slot1values,slot2values,slot3values,slot4values,slot5values FROM _timescaledb_internal.get_chunk_colstats(relid => 'public.telemetries')
@NunoFilipeSantos NunoFilipeSantos added multinode segfault Segmentation fault labels Oct 25, 2021
@nikkhils
Copy link
Contributor

@tanglebones will it be possible for you to try out the latest 2.4.2 release of timescaledb with Postgres 13.4?

@nikkhils
Copy link
Contributor

@tanglebones the crash is occurring only when postgis extension is used in the table. I replaced the geography field type by text and subsequent ANALYZE does not crash.

@nikkhils
Copy link
Contributor

The crash seems to be coming from the collect_colstat_slots function. It's kinda difficult to get any further information from the docker setup..

@nikkhils
Copy link
Contributor

nikkhils commented Oct 26, 2021 •
edited
Loading

I managed to reproduced this locally. The stack trace is as below:

frame #3: 0x000000010c1b802d postgres`ExceptionalCondition(conditionName="HeapTupleIsValid(operator_tuple)", errorType="FailedAssertion", fileName="/Users/Nikhils/Work/TS/timescaledb/tsl/src/chunk_api.c", lineNumber=668) at assert.c:67:2
    frame #4: 0x000000010cadb9af timescaledb-tsl-2.5.0-dev.so`convert_op_oid_to_strings(op_id=0, result_strings=0x00007ffee41b9ad0) at chunk_api.c:668:2
  * frame #5: 0x000000010cadb4cb timescaledb-tsl-2.5.0-dev.so`collect_colstat_slots(tuple=0x00007faff8186cc8, formdata=0x00007faff8186d10, values=0x00007ffee41b9bf0, nulls=0x00007ffee41b9bd0) at chunk_api.c:767:3
    frame #6: 0x000000010cadb312 timescaledb-tsl-2.5.0-dev.so`chunk_get_single_colstats_tuple(chunk=0x00007fb001049938, column=5, tupdesc=0x00007fafc1016440) at chunk_api.c:909:2
    frame #7: 0x000000010cadad7b timescaledb-tsl-2.5.0-dev.so`chunk_api_fetch_next_colstats_tuple(funcctx=0x00007fb002831d20) at chunk_api.c:1465:11
    frame #8: 0x000000010cad8aa3 timescaledb-tsl-2.5.0-dev.so`chunk_api_get_chunk_stats(fcinfo=0x00007fb001030520, col_stats=true) at chunk_api.c:1640:22

The assert that is getting hit is:

Assert(HeapTupleIsValid(operator_tuple));

@tanglebones
Copy link
Author

Tried with PostgreSQL: 13.4, PostGIS: 3.1.4, TimescaleDB: 2.4.2 and got the segfault. (Sorry, haven't had time to update the branch just yet with a repeatable test case). Removing PostGIS does prevent the segfault.

@nikkhils nikkhils self-assigned this Oct 27, 2021
@nikkhils
Copy link
Contributor

I am working on a fix for this and it will a part of the upcoming release.

nikkhils added a commit to nikkhils/timescaledb that referenced this issue Oct 27, 2021
@nikkhils
Fix ANALYZE crash with custom types
edc8f8d 
Extensions like PostGIS introduce custom datatypes for columns. They
also have custom statistics for such custom types. We already state in
our comments that stats for custom types are not supported but it was
never tested. Fix that now.

Fixes timescale#3733
@nikkhils nikkhils mentioned this issue Oct 27, 2021
Merged
nikkhils added a commit to nikkhils/timescaledb that referenced this issue Oct 27, 2021
@nikkhils
Fix ANALYZE crash with custom types
af7f6ea 
Extensions like PostGIS introduce custom datatypes for columns. They
also have custom statistics for such custom types. We already state in
our comments that stats for custom types are not supported but it was
never tested. Fix that now.

Fixes timescale#3733
nikkhils added a commit to nikkhils/timescaledb that referenced this issue Oct 27, 2021
@nikkhils
Fix ANALYZE crash with custom types
e4a174e 
Extensions like PostGIS introduce custom datatypes for columns. They
also have custom statistics for such custom types. We already state in
our comments that stats for custom types are not supported but it was
never tested. Fix that now.

Fixes timescale#3733
nikkhils added a commit that referenced this issue Oct 27, 2021
@nikkhils
Fix ANALYZE crash with custom types
6db012d 
Extensions like PostGIS introduce custom datatypes for columns. They
also have custom statistics for such custom types. We already state in
our comments that stats for custom types are not supported but it was
never tested. Fix that now.

Fixes #3733
fabriziomello added a commit to fabriziomello/timescaledb that referenced this issue Oct 27, 2021
@fabriziomello
Release 2.5.0
6d308d9 
This release adds major new features since the 2.4.2 release.
We deem it moderate priority for upgrading.

This release includes these noteworthy features:

* Continuous Aggregates for Distributed Hypertables
* Support for PostgreSQL 14
* Experimental: Support for timezones in `time_bucket_ng()`, including
the `origin` argument

This release also includes several bug fixes.

**Features**
* timescale#3034 Add support for PostgreSQL 14
* timescale#3435 Add continuous aggregates for distributed hypertables
* timescale#3505 Add support for timezones in `time_bucket_ng()`
* timescale#3598 Improve evaluation of stable functions such as now() on access node
* timescale#3717 Support transparent decompression on individual chunks

**Bugfixes**
* timescale#3580 Fix memory context bug executing TRUNCATE
* timescale#3592 Allow alter column type on distributed hypertable
* timescale#3618 Fix execution of refresh_caggs from user actions
* timescale#3625 Add shared dependencies when creating chunk
* timescale#3626 Fix memory context bug executing TRUNCATE
* timescale#3627 Schema qualify UDTs in multi-node
* timescale#3638 Allow owner change of a data node
* timescale#3654 Fix index attnum mapping in reorder_chunk
* timescale#3661 Fix SkipScan path generation with constant DISTINCT column
* timescale#3667 Fix compress_policy for multi txn handling
* timescale#3673 Fix distributed hypertable DROP within a procedure
* timescale#3701 Allow anyone to use size utilities on distributed hypertables
* timescale#3708 Fix crash in get_aggsplit
* timescale#3709 Fix ordered append pathkey check
* timescale#3712 Fix GRANT/REVOKE ALL IN SCHEMA handling
* timescale#3724 Fix inserts into compressed chunks on hypertables with caggs
* timescale#3727 Fix DirectFunctionCall crash in distributed_exec
* timescale#3728 Fix SkipScan with varchar column
* timescale#3733 Fix ANALYZE crash with custom statistics for custom types
* timescale#3747 Always reset expr context in DecompressChunk

**Thanks**
* @binakot and @sebvett for reporting an issue with DISTINCT queries
* @hardikm10, @DavidPavlicek and @pafiti for reporting bugs on TRUNCATE
* @mjf for reporting an issue with ordered append and JOINs
* @phemmer for reporting the issues on multinode with aggregate queries and evaluation of now()
* @abolognino for reporting an issue with INSERTs into compressed hypertables that have cagg
* @tanglebones for reporting the ANALYZE crash with custom types on multinode
@fabriziomello fabriziomello mentioned this issue Oct 27, 2021
Merged
fabriziomello added a commit to fabriziomello/timescaledb that referenced this issue Oct 27, 2021
@fabriziomello
Release 2.5.0
a9a5de6 
This release adds major new features since the 2.4.2 release.
We deem it moderate priority for upgrading.

This release includes these noteworthy features:

* Continuous Aggregates for Distributed Hypertables
* Support for PostgreSQL 14
* Experimental: Support for timezones in `time_bucket_ng()`, including
the `origin` argument

This release also includes several bug fixes.

**Features**
* timescale#3034 Add support for PostgreSQL 14
* timescale#3435 Add continuous aggregates for distributed hypertables
* timescale#3505 Add support for timezones in `time_bucket_ng()`

**Bugfixes**
* timescale#3580 Fix memory context bug executing TRUNCATE
* timescale#3592 Allow alter column type on distributed hypertable
* timescale#3598 Improve evaluation of stable functions such as now() on access
node
* timescale#3618 Fix execution of refresh_caggs from user actions
* timescale#3625 Add shared dependencies when creating chunk
* timescale#3626 Fix memory context bug executing TRUNCATE
* timescale#3627 Schema qualify UDTs in multi-node
* timescale#3638 Allow owner change of a data node
* timescale#3654 Fix index attnum mapping in reorder_chunk
* timescale#3661 Fix SkipScan path generation with constant DISTINCT column
* timescale#3667 Fix compress_policy for multi txn handling
* timescale#3673 Fix distributed hypertable DROP within a procedure
* timescale#3701 Allow anyone to use size utilities on distributed hypertables
* timescale#3708 Fix crash in get_aggsplit
* timescale#3709 Fix ordered append pathkey check
* timescale#3712 Fix GRANT/REVOKE ALL IN SCHEMA handling
* timescale#3717 Support transparent decompression on individual chunks
* timescale#3724 Fix inserts into compressed chunks on hypertables with caggs
* timescale#3727 Fix DirectFunctionCall crash in distributed_exec
* timescale#3728 Fix SkipScan with varchar column
* timescale#3733 Fix ANALYZE crash with custom statistics for custom types
* timescale#3747 Always reset expr context in DecompressChunk

**Thanks**
* @binakot and @sebvett for reporting an issue with DISTINCT queries
* @hardikm10, @DavidPavlicek and @pafiti for reporting bugs on TRUNCATE
* @mjf for reporting an issue with ordered append and JOINs
* @phemmer for reporting the issues on multinode with aggregate queries and evaluation of now()
* @abolognino for reporting an issue with INSERTs into compressed hypertables that have cagg
* @tanglebones for reporting the ANALYZE crash with custom types on multinode
fabriziomello added a commit that referenced this issue Oct 27, 2021
@fabriziomello
Release 2.5.0
8925dd8 
This release adds major new features since the 2.4.2 release.
We deem it moderate priority for upgrading.

This release includes these noteworthy features:

* Continuous Aggregates for Distributed Hypertables
* Support for PostgreSQL 14
* Experimental: Support for timezones in `time_bucket_ng()`, including
the `origin` argument

This release also includes several bug fixes.

**Features**
* #3034 Add support for PostgreSQL 14
* #3435 Add continuous aggregates for distributed hypertables
* #3505 Add support for timezones in `time_bucket_ng()`

**Bugfixes**
* #3580 Fix memory context bug executing TRUNCATE
* #3592 Allow alter column type on distributed hypertable
* #3598 Improve evaluation of stable functions such as now() on access
node
* #3618 Fix execution of refresh_caggs from user actions
* #3625 Add shared dependencies when creating chunk
* #3626 Fix memory context bug executing TRUNCATE
* #3627 Schema qualify UDTs in multi-node
* #3638 Allow owner change of a data node
* #3654 Fix index attnum mapping in reorder_chunk
* #3661 Fix SkipScan path generation with constant DISTINCT column
* #3667 Fix compress_policy for multi txn handling
* #3673 Fix distributed hypertable DROP within a procedure
* #3701 Allow anyone to use size utilities on distributed hypertables
* #3708 Fix crash in get_aggsplit
* #3709 Fix ordered append pathkey check
* #3712 Fix GRANT/REVOKE ALL IN SCHEMA handling
* #3717 Support transparent decompression on individual chunks
* #3724 Fix inserts into compressed chunks on hypertables with caggs
* #3727 Fix DirectFunctionCall crash in distributed_exec
* #3728 Fix SkipScan with varchar column
* #3733 Fix ANALYZE crash with custom statistics for custom types
* #3747 Always reset expr context in DecompressChunk

**Thanks**
* @binakot and @sebvett for reporting an issue with DISTINCT queries
* @hardikm10, @DavidPavlicek and @pafiti for reporting bugs on TRUNCATE
* @mjf for reporting an issue with ordered append and JOINs
* @phemmer for reporting the issues on multinode with aggregate queries and evaluation of now()
* @abolognino for reporting an issue with INSERTs into compressed hypertables that have cagg
* @tanglebones for reporting the ANALYZE crash with custom types on multinode
fabriziomello added a commit that referenced this issue Oct 27, 2021
@fabriziomello
Release 2.5.0
ca83433 
This release adds major new features since the 2.4.2 release.
We deem it moderate priority for upgrading.

This release includes these noteworthy features:

* Continuous Aggregates for Distributed Hypertables
* Support for PostgreSQL 14
* Experimental: Support for timezones in `time_bucket_ng()`, including
the `origin` argument

This release also includes several bug fixes.

**Features**
* #3034 Add support for PostgreSQL 14
* #3435 Add continuous aggregates for distributed hypertables
* #3505 Add support for timezones in `time_bucket_ng()`

**Bugfixes**
* #3580 Fix memory context bug executing TRUNCATE
* #3592 Allow alter column type on distributed hypertable
* #3598 Improve evaluation of stable functions such as now() on access
node
* #3618 Fix execution of refresh_caggs from user actions
* #3625 Add shared dependencies when creating chunk
* #3626 Fix memory context bug executing TRUNCATE
* #3627 Schema qualify UDTs in multi-node
* #3638 Allow owner change of a data node
* #3654 Fix index attnum mapping in reorder_chunk
* #3661 Fix SkipScan path generation with constant DISTINCT column
* #3667 Fix compress_policy for multi txn handling
* #3673 Fix distributed hypertable DROP within a procedure
* #3701 Allow anyone to use size utilities on distributed hypertables
* #3708 Fix crash in get_aggsplit
* #3709 Fix ordered append pathkey check
* #3712 Fix GRANT/REVOKE ALL IN SCHEMA handling
* #3717 Support transparent decompression on individual chunks
* #3724 Fix inserts into compressed chunks on hypertables with caggs
* #3727 Fix DirectFunctionCall crash in distributed_exec
* #3728 Fix SkipScan with varchar column
* #3733 Fix ANALYZE crash with custom statistics for custom types
* #3747 Always reset expr context in DecompressChunk

**Thanks**
* @binakot and @sebvett for reporting an issue with DISTINCT queries
* @hardikm10, @DavidPavlicek and @pafiti for reporting bugs on TRUNCATE
* @mjf for reporting an issue with ordered append and JOINs
* @phemmer for reporting the issues on multinode with aggregate queries
and evaluation of now()
* @abolognino for reporting an issue with INSERTs into compressed
hypertables that have cagg
* @tanglebones for reporting the ANALYZE crash with custom types on
multinode
fabriziomello added a commit that referenced this issue Oct 27, 2021
@fabriziomello
Release 2.5.0
6245bcd 
This release adds major new features since the 2.4.2 release.
We deem it moderate priority for upgrading.

This release includes these noteworthy features:

* Continuous Aggregates for Distributed Hypertables
* Support for PostgreSQL 14
* Experimental: Support for timezones in `time_bucket_ng()`, including
the `origin` argument

This release also includes several bug fixes.

**Features**
* #3034 Add support for PostgreSQL 14
* #3435 Add continuous aggregates for distributed hypertables
* #3505 Add support for timezones in `time_bucket_ng()`

**Bugfixes**
* #3580 Fix memory context bug executing TRUNCATE
* #3592 Allow alter column type on distributed hypertable
* #3598 Improve evaluation of stable functions such as now() on access
node
* #3618 Fix execution of refresh_caggs from user actions
* #3625 Add shared dependencies when creating chunk
* #3626 Fix memory context bug executing TRUNCATE
* #3627 Schema qualify UDTs in multi-node
* #3638 Allow owner change of a data node
* #3654 Fix index attnum mapping in reorder_chunk
* #3661 Fix SkipScan path generation with constant DISTINCT column
* #3667 Fix compress_policy for multi txn handling
* #3673 Fix distributed hypertable DROP within a procedure
* #3701 Allow anyone to use size utilities on distributed hypertables
* #3708 Fix crash in get_aggsplit
* #3709 Fix ordered append pathkey check
* #3712 Fix GRANT/REVOKE ALL IN SCHEMA handling
* #3717 Support transparent decompression on individual chunks
* #3724 Fix inserts into compressed chunks on hypertables with caggs
* #3727 Fix DirectFunctionCall crash in distributed_exec
* #3728 Fix SkipScan with varchar column
* #3733 Fix ANALYZE crash with custom statistics for custom types
* #3747 Always reset expr context in DecompressChunk

**Thanks**
* @binakot and @sebvett for reporting an issue with DISTINCT queries
* @hardikm10, @DavidPavlicek and @pafiti for reporting bugs on TRUNCATE
* @mjf for reporting an issue with ordered append and JOINs
* @phemmer for reporting the issues on multinode with aggregate queries
and evaluation of now()
* @abolognino for reporting an issue with INSERTs into compressed
hypertables that have cagg
* @tanglebones for reporting the ANALYZE crash with custom types on
multinode
fabriziomello added a commit that referenced this issue Oct 27, 2021
@fabriziomello
Release 2.5.0
e6f137f 
This release adds major new features since the 2.4.2 release.
We deem it moderate priority for upgrading.

This release includes these noteworthy features:

* Continuous Aggregates for Distributed Hypertables
* Support for PostgreSQL 14
* Experimental: Support for timezones in `time_bucket_ng()`, including
the `origin` argument

This release also includes several bug fixes.

**Features**
* #3034 Add support for PostgreSQL 14
* #3435 Add continuous aggregates for distributed hypertables
* #3505 Add support for timezones in `time_bucket_ng()`

**Bugfixes**
* #3580 Fix memory context bug executing TRUNCATE
* #3592 Allow alter column type on distributed hypertable
* #3598 Improve evaluation of stable functions such as now() on access
node
* #3618 Fix execution of refresh_caggs from user actions
* #3625 Add shared dependencies when creating chunk
* #3626 Fix memory context bug executing TRUNCATE
* #3627 Schema qualify UDTs in multi-node
* #3638 Allow owner change of a data node
* #3654 Fix index attnum mapping in reorder_chunk
* #3661 Fix SkipScan path generation with constant DISTINCT column
* #3667 Fix compress_policy for multi txn handling
* #3673 Fix distributed hypertable DROP within a procedure
* #3701 Allow anyone to use size utilities on distributed hypertables
* #3708 Fix crash in get_aggsplit
* #3709 Fix ordered append pathkey check
* #3712 Fix GRANT/REVOKE ALL IN SCHEMA handling
* #3717 Support transparent decompression on individual chunks
* #3724 Fix inserts into compressed chunks on hypertables with caggs
* #3727 Fix DirectFunctionCall crash in distributed_exec
* #3728 Fix SkipScan with varchar column
* #3733 Fix ANALYZE crash with custom statistics for custom types
* #3747 Always reset expr context in DecompressChunk

**Thanks**
* @binakot and @sebvett for reporting an issue with DISTINCT queries
* @hardikm10, @DavidPavlicek and @pafiti for reporting bugs on TRUNCATE
* @mjf for reporting an issue with ordered append and JOINs
* @phemmer for reporting the issues on multinode with aggregate queries
and evaluation of now()
* @abolognino for reporting an issue with INSERTs into compressed
hypertables that have cagg
* @tanglebones for reporting the ANALYZE crash with custom types on
multinode
@tanglebones
Copy link
Author

@nikkhils Thanks for the quick replies and fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nikkhils nikkhils

Labels
multinode postgis segfault Segmentation fault
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix ANALYZE crash with custom types nikkhils/timescaledb
3 participants
@tanglebones @nikkhils @NunoFilipeSantos