Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segfault in 1.6.1 in complex query #1841

Closed
akamensky opened this issue Apr 22, 2020 · 18 comments · Fixed by #2256
Closed

Segfault in 1.6.1 in complex query #1841

akamensky opened this issue Apr 22, 2020 · 18 comments · Fixed by #2256
Assignees
Labels
bug segfault Segmentation fault

Comments

@akamensky
Copy link

Relevant system information:

  • OS: Centos 7.5
  • PostgreSQL version: 10.9
  • TimescaleDB version: 1.6.1
  • Installation method: yum

Describe the bug
Segfault when running rather complex query:

segfault at 0 ip 00007f4003baddc8 sp 00007fff41c475f0 error 4 in timescaledb-1.6.1.so[7f4003b69000+65000]
  • See query and table structure below
  • Crash only happens when the specific WHERE clause is used AND when return set is not empty (that is query result would be not 0 rows)
  • The clause that is casing segfault is one of "col19" in ('123') or "col19" = '123' (only happens on col19)

Table definition:

# \d+ segfaulttable
                                                      Table "public.segfaulttable"
            Column             |            Type             | Collation | Nullable | Default | Storage  | Stats target | Description 
-------------------------------+-----------------------------+-----------+----------+---------+----------+--------------+-------------
 col1                          | numeric                     |           | not null |         | main     |              | 
 col2                          | double precision            |           | not null |         | plain    |              | 
 col3                          | double precision            |           | not null |         | plain    |              | 
 col4                          | text                        |           | not null |         | extended |              | 
 col5                          | text                        |           | not null |         | extended |              | 
 col6                          | double precision            |           | not null |         | plain    |              | 
 col7                          | double precision            |           | not null |         | plain    |              | 
 col8                          | date                        |           | not null |         | plain    |              | 
 createTsMillis                | timestamp without time zone |           | not null |         | plain    |              | 
 col9                          | double precision            |           | not null |         | plain    |              | 
 col10                         | double precision            |           | not null |         | plain    |              | 
 col11                         | numeric                     |           | not null |         | main     |              | 
 col12                         | double precision            |           | not null |         | plain    |              | 
 col13                         | numeric                     |           | not null |         | main     |              | 
 col14                         | double precision            |           | not null |         | plain    |              | 
 col15                         | double precision            |           | not null |         | plain    |              | 
 createTs                      | bigint                      |           | not null |         | plain    |              | 
 col16                         | bigint                      |           | not null |         | plain    |              | 
 col17                         | text                        |           | not null |         | extended |              | 
 col18                         | date                        |           | not null |         | plain    |              | 
 col19                         | text                        |           | not null |         | extended |              | 
 col20                         | bigint                      |           | not null |         | plain    |              | 
 col21                         | numeric                     |           | not null |         | main     |              | 
 col22                         | numeric                     |           | not null |         | main     |              | 
 col23                         | numeric                     |           | not null |         | main     |              | 
 createTsMicros                | bigint                      |           | not null |         | plain    |              | 
 col24                         | text                        |           | not null |         | extended |              | 
 col25                         | double precision            |           | not null |         | plain    |              | 
 col26                         | numeric                     |           | not null |         | main     |              | 
 col27                         | double precision            |           | not null |         | plain    |              | 
 col28                         | double precision            |           | not null |         | plain    |              | 
 col29                         | double precision            |           | not null |         | plain    |              | 
 col30                         | text                        |           | not null |         | extended |              | 
 col31                         | double precision            |           | not null |         | plain    |              | 
 col32                         | double precision            |           | not null |         | plain    |              | 
 col33                         | text                        |           |          |         | extended |              | 
 col34                         | numeric                     |           | not null |         | main     |              | 
 col35                         | numeric                     |           | not null |         | main     |              | 
 col36                         | double precision            |           | not null |         | plain    |              | 
 col37                         | numeric                     |           | not null |         | main     |              | 
Indexes:
    "segfaulttable_createTsMillis_idx" btree ("createTsMillis" DESC)
Triggers:
    ts_insert_blocker BEFORE INSERT ON pwitickdata FOR EACH ROW EXECUTE PROCEDURE _timescaledb_internal.insert_blocker()
Child tables: _timescaledb_internal._hyper_41_188954_chunk,
              _timescaledb_internal._hyper_41_188955_chunk,
              _timescaledb_internal._hyper_41_188956_chunk,
              _timescaledb_internal._hyper_41_188957_chunk,
              _timescaledb_internal._hyper_41_193080_chunk,
              _timescaledb_internal._hyper_41_193081_chunk,
              _timescaledb_internal._hyper_41_193082_chunk,
              _timescaledb_internal._hyper_41_193083_chunk,
              _timescaledb_internal._hyper_41_195707_chunk,
              _timescaledb_internal._hyper_41_195708_chunk,
              _timescaledb_internal._hyper_41_195709_chunk,
              _timescaledb_internal._hyper_41_195710_chunk

The query:

SELECT 
    time_bucket('1 seconds', "createTsMillis") as time_bucket,
    "col19", 
    LAST("col17","createTsMillis") as col17_last,
    LAST("col35","createTsMillis") as col35_last,
    AVG("col25") as col25_avg, 
    AVG("col6") as col6_avg,
    AVG("col32") as col32_avg, 
    AVG("col10") as col10_avg,
    (last("col16","createTsMillis") - first("col16","createTsMillis")) / last("col11","createTsMillis") as col16_period,
    first("col13","createTsMillis") as col13_first,
    (last("col13","createTsMillis") - first("col13","createTsMillis")) as col13_change,
    count("createTsMillis") as snap_count,
    last("col16","createTsMillis") as col16_last
FROM segfaulttable
WHERE 
    "col19" = '123'
    AND (("createTsMillis" >= '2020-04-22 01:30:00' AND "createTsMillis" <= '2020-04-22 04:00:00') OR ("createTsMillis" >= '2020-04-22 05:00:00' AND "createTsMillis" <= '2020-04-22 08:00:00'))
    AND "col10" <> 'NaN'
    AND "col32" <> 'NaN'
GROUP BY "col19", time_bucket
ORDER BY time_bucket;
@pmwkaa pmwkaa self-assigned this Apr 23, 2020
@akamensky
Copy link
Author

FWIW, dropping ORDER BY time_bucket also stops crashing. But I reckon it still needs to be fixed perhaps.

@pmwkaa
Copy link
Contributor

pmwkaa commented Apr 29, 2020

Hey @akamensky, I was not able to reproduce so far using self-generated data. Could you please provide minimal dataset (or query to generate it) which can be used for easier testing, thanks

@akamensky
Copy link
Author

@pmwkaa Don't think that would be allowed by company policy. You should be able to generate data based on types in table definition I provided. Also possibly the conditions I added in the top post could help (i.e. no crash on 0 rows result).

@pmwkaa
Copy link
Contributor

pmwkaa commented Apr 30, 2020

@akamensky understood, no worries just had an impression that you used generated data for this test case. I was able to generate some data to match the SELECT query, but did not experience any crash so far, likely I am missing something:

INSERT INTO segfaulttable ("createTsMillis", col19, col10, col32, col25, col6)
SELECT time, '123', random(), random(), random(), random() FROM
generate_series('2020-04-22 01:30:00'::timestamptz,
                '2020-04-22 04:00:00'::timestamptz,
                '1 millisecond') as time;

Could you please show the backtrace of the core dump, in case if there are some meaningful information as well?

@akamensky
Copy link
Author

That should be doable. We are on public holiday at the moment until Monday. I’ll be able to generate core dump on Monday earliest. Will add that then.

@akamensky
Copy link
Author

@pmwkaa I think i forgot to mention one important setup detail here -- the crash happens on read-only replica host (using postgresql native streaming replication). That may be a factor as well. Have not tried this on primary server (and won't be able to try it due to stability requirements).

@pmwkaa
Copy link
Contributor

pmwkaa commented May 4, 2020

@akamensky interesting, thanks for the details

@pmwkaa
Copy link
Contributor

pmwkaa commented May 4, 2020

I've tried to create a master-replica setup and reproduce the issue on the replica host. Unfortunately everything seems to be working so far, so any new information would be useful.

@pmwkaa
Copy link
Contributor

pmwkaa commented May 11, 2020

Any luck yet?

@akamensky
Copy link
Author

Loaded symbols for /usr/pgsql-10/lib/timescaledb.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
0x00007f7dcd5ba163 in __epoll_wait_nocancel () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install postgresql10-server-10.9-1PGDG.rhel7.x86_64
(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x00007f6d1971adc8 in ts_chunk_append_path_create () from /usr/pgsql-10/lib/timescaledb-1.6.1.so

@akamensky
Copy link
Author

Unfortunately ALL binaries provided by you are stripped of debugging symbols and no separate debugging symbol files are provided. I do not have the capability to compile and debug your code manually.

gdb -c core.4538 
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
[New LWP 4538]
Core was generated by `postgres: postgres tickdata_hk [local] SELECT'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f6d1971adc8 in ?? ()
(gdb) symbol-file /usr/pgsql-10/lib/timescaledb-1.6.1.so
Reading symbols from /usr/pgsql-10/lib/timescaledb-1.6.1.so...Missing separate debuginfo for /usr/pgsql-10/lib/timescaledb-1.6.1.so

@akamensky
Copy link
Author

Full output from GDB session:

[root@hkdb05 dump]# gdb --pid=9484
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 9484
Reading symbols from /usr/pgsql-10/bin/postgres...Reading symbols from /usr/pgsql-10/bin/postgres...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libxml2.so.2...Reading symbols from /lib64/libxml2.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libxml2.so.2
Reading symbols from /lib64/libpam.so.0...Reading symbols from /lib64/libpam.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libpam.so.0
Reading symbols from /lib64/libssl.so.10...Reading symbols from /lib64/libssl.so.10...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libssl.so.10
Reading symbols from /lib64/libcrypto.so.10...Reading symbols from /lib64/libcrypto.so.10...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypto.so.10
Reading symbols from /lib64/libgssapi_krb5.so.2...Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libldap-2.4.so.2...Reading symbols from /lib64/libldap-2.4.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libldap-2.4.so.2
Reading symbols from /lib64/libicui18n.so.50...Reading symbols from /lib64/libicui18n.so.50...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libicui18n.so.50
Reading symbols from /lib64/libicuuc.so.50...Reading symbols from /lib64/libicuuc.so.50...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libicuuc.so.50
Reading symbols from /lib64/libsystemd.so.0...Reading symbols from /lib64/libsystemd.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libsystemd.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libz.so.1...Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/liblzma.so.5...Reading symbols from /lib64/liblzma.so.5...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/liblzma.so.5
Reading symbols from /lib64/libaudit.so.1...Reading symbols from /lib64/libaudit.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libaudit.so.1
Reading symbols from /lib64/libkrb5.so.3...Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libkrb5support.so.0...Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/liblber-2.4.so.2...Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/liblber-2.4.so.2
Reading symbols from /lib64/libsasl2.so.3...Reading symbols from /lib64/libsasl2.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libsasl2.so.3
Reading symbols from /lib64/libssl3.so...Reading symbols from /lib64/libssl3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libssl3.so
Reading symbols from /lib64/libsmime3.so...Reading symbols from /lib64/libsmime3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libsmime3.so
Reading symbols from /lib64/libnss3.so...Reading symbols from /lib64/libnss3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss3.so
Reading symbols from /lib64/libnssutil3.so...Reading symbols from /lib64/libnssutil3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnssutil3.so
Reading symbols from /lib64/libplds4.so...Reading symbols from /lib64/libplds4.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libplds4.so
Reading symbols from /lib64/libplc4.so...Reading symbols from /lib64/libplc4.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libplc4.so
Reading symbols from /lib64/libnspr4.so...Reading symbols from /lib64/libnspr4.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnspr4.so
Reading symbols from /lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libicudata.so.50...Reading symbols from /lib64/libicudata.so.50...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libicudata.so.50
Reading symbols from /lib64/libcap.so.2...Reading symbols from /lib64/libcap.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcap.so.2
Reading symbols from /lib64/libselinux.so.1...Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/liblz4.so.1...Reading symbols from /lib64/liblz4.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/liblz4.so.1
Reading symbols from /lib64/libgcrypt.so.11...Reading symbols from /lib64/libgcrypt.so.11...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcrypt.so.11
Reading symbols from /lib64/libgpg-error.so.0...Reading symbols from /lib64/libgpg-error.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libgpg-error.so.0
Reading symbols from /lib64/libdw.so.1...Reading symbols from /lib64/libdw.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libdw.so.1
Reading symbols from /lib64/libcap-ng.so.0...Reading symbols from /lib64/libcap-ng.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcap-ng.so.0
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libattr.so.1...Reading symbols from /lib64/libattr.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libattr.so.1
Reading symbols from /lib64/libpcre.so.1...Reading symbols from /lib64/libpcre.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libpcre.so.1
Reading symbols from /lib64/libelf.so.1...Reading symbols from /lib64/libelf.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libelf.so.1
Reading symbols from /lib64/libbz2.so.1...Reading symbols from /lib64/libbz2.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libbz2.so.1
Reading symbols from /lib64/libfreebl3.so...Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /usr/pgsql-10/lib/timescaledb.so...Reading symbols from /usr/pgsql-10/lib/timescaledb.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /usr/pgsql-10/lib/timescaledb.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
0x00007f7dcd5ba163 in __epoll_wait_nocancel () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install postgresql10-server-10.9-1PGDG.rhel7.x86_64
(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x00007f6d1971adc8 in ts_chunk_append_path_create () from /usr/pgsql-10/lib/timescaledb-1.6.1.so
(gdb) thread apply all bt full

Thread 1 (Thread 0x7f7dcfe4a8c0 (LWP 9484)):
#0  0x00007f6d1971adc8 in ts_chunk_append_path_create () from /usr/pgsql-10/lib/timescaledb-1.6.1.so
No symbol table info available.
#1  0x00007f6d1970bdee in timescaledb_set_rel_pathlist () from /usr/pgsql-10/lib/timescaledb-1.6.1.so
No symbol table info available.
#2  0x000000000065ffe2 in set_rel_pathlist ()
No symbol table info available.
#3  0x0000000000660b80 in make_one_rel ()
No symbol table info available.
#4  0x000000000067d33e in query_planner ()
No symbol table info available.
#5  0x000000000067fbb3 in grouping_planner ()
No symbol table info available.
#6  0x0000000000682885 in subquery_planner ()
No symbol table info available.
#7  0x000000000068385f in standard_planner ()
No symbol table info available.
#8  0x00007f6d1970b8c5 in timescaledb_planner () from /usr/pgsql-10/lib/timescaledb-1.6.1.so
No symbol table info available.
#9  0x000000000071b05e in pg_plan_query ()
No symbol table info available.
#10 0x000000000071b11e in pg_plan_queries ()
No symbol table info available.
#11 0x000000000071b5ca in exec_simple_query ()
No symbol table info available.
#12 0x000000000071c77c in PostgresMain ()
No symbol table info available.
#13 0x000000000047b326 in ServerLoop ()
No symbol table info available.
#14 0x00000000006b2dcd in PostmasterMain ()
No symbol table info available.
#15 0x000000000047c14f in main ()
No symbol table info available.
(gdb)

@pmwkaa
Copy link
Contributor

pmwkaa commented May 27, 2020

@akamensky Thanks for the debug info. Apparently crash happened in the optimization part. Right now I would suggest try to disable it by running:

SET timescaledb.enable_chunk_append to OFF

Otherwise, maybe there is a chance to test it against the latest version which is 1.7.1?

@akamensky
Copy link
Author

@pmwkaa since our current postgres is 10.9 and support for 10.x is dropped starting 1.7 we could not upgrade to 1.7 (though some desired features are in that release).

We are looking to upgrade to latest postgres 12 on those hosts in next few weeks (but will likely be some time until then).

Meantime does disabling this option have any effect on performance and/or returned result set? Since this is only happening in our production environment (could not reproduce in test environment), we would like to avoid any impact there.

@akamensky
Copy link
Author

akamensky commented Jun 10, 2020

We are in the middle of upgrading to 1.7.1. Meantime other suggestions did not change the outcome (still segfault). I am not sure whether the issue will be reproducible in 1.7.1, because binary upgrade is not possible due to #1973

@mfreed
Copy link
Member

mfreed commented Jul 27, 2020

Hi @akamensky Are you able to reproduce it in 1.7.1?

@akamensky
Copy link
Author

@mfreed we are currently taking #1986 as higher priority for us and still waiting on feedback there (should be reopened though I can't reopen). Once that one is dealt with and we stop hourly segfaults then we can retest this issue in 1.7.2

@nexces
Copy link

nexces commented Aug 11, 2020

@mfreed
We were able to reproduce this bug on version 1.7.2.

Query causing the problem:

SELECT
    date_trunc('hour', created) AS czas,
    count(*) AS liczba
  FROM measurement
  WHERE created BETWEEN '2020-08-10 06:00:00' AND '2020-08-10 10:00:00' OR created BETWEEN '2020-08-11 06:00:00' AND '2020-08-11 10:00:00' 
  GROUP BY date_trunc('hour', created)
  ORDER BY date_trunc('hour', created); -- ORDER BY 1

Table structure:

treesat_storage=# \d measurement
                      Table "public.measurement"
  Column  |           Type           | Collation | Nullable | Default
----------+--------------------------+-----------+----------+---------
 created  | timestamp with time zone |           | not null |
 inserted | timestamp with time zone |           | not null | now()
 id       | text                     |           | not null |
 value    | jsonb                    |           | not null |
Indexes:
    "pk_measurement" PRIMARY KEY, btree (id, created)
    "measurement_created_idx" btree (created DESC)
Triggers:
    ts_insert_blocker BEFORE INSERT ON measurement FOR EACH ROW EXECUTE FUNCTION _timescaledb_internal.insert_blocker()
Number of child tables: 7794 (Use \d+ to list them.)
GNU gdb (Debian 8.2.1-2+b3) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 11928
Reading symbols from /usr/lib/postgresql/12/bin/postgres...Reading symbols from /usr/lib/debug/.build-id/25/57795aed00d014a885ff1e9616875d902b03ad.debug...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols from /usr/lib/debug/.build-id/e9/1114987a0147bd050addbd591eb8994b29f4b3.debug...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Reading symbols from /lib/x86_64-linux-gnu/libxml2.so.2...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libpam.so.0...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libssl.so.1.1...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libcrypto.so.1.1...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...Reading symbols from /usr/lib/debug/.build-id/5d/cf98ad684962be494af28a1051793fd39e4ebc.debug...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading symbols from /usr/lib/debug/.build-id/d3/583c742dd47aaa860c5ae0c0c5bdbcd2d54f61.debug...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...Reading symbols from /usr/lib/debug/.build-id/88/5dda4b4a5cea600e7b5b98c1ad86996c8d2299.debug...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libldap_r-2.4.so.2...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libicui18n.so.63...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libicuuc.so.63...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libsystemd.so.0...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug/.build-id/18/b9a9a8c523e5cfe5b5d946d605d09242f09798.debug...done.
done.
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/.build-id/f2/5dfd7b95be4ba386fd71080accae8c0732b711.debug...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libicudata.so.63...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/liblzma.so.5...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libaudit.so.1...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libkrb5.so.3...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libk5crypto.so.3...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libcom_err.so.2...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libkrb5support.so.0...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libkeyutils.so.1...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libresolv.so.2...Reading symbols from /usr/lib/debug/.build-id/02/6c3ba167f64f631eb8781fca2269fbc2ee7ca5.debug...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/liblber-2.4.so.2...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libsasl2.so.2...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libgnutls.so.30...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/liblz4.so.1...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libgcrypt.so.20...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libcap-ng.so.0...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libp11-kit.so.0...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libidn2.so.0...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libunistring.so.2...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libtasn1.so.6...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libnettle.so.6...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libhogweed.so.4...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libgmp.so.10...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libgpg-error.so.0...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libffi.so.6...(no debugging symbols found)...done.
Reading symbols from /usr/lib/postgresql/12/lib/timescaledb.so...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...Reading symbols from /usr/lib/debug/.build-id/4b/ff8b782e1602c596e856bdef06e642e50e7fa7.debug...done.
done.
Reading symbols from /usr/lib/postgresql/12/lib/timescaledb-1.7.2.so...done.
Reading symbols from /usr/lib/postgresql/12/lib/timescaledb-tsl-1.7.2.so...done.
0x00007f533b6f67b7 in epoll_wait (epfd=3, events=0x55e7e0118840, maxevents=maxevents@entry=1, timeout=timeout@entry=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
30      ../sysdeps/unix/sysv/linux/epoll_wait.c: No such file or directory.
(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
ts_chunk_append_path_create (root=root@entry=0x55e7e01ac958, rel=rel@entry=0x55e7e01ab738, ht=ht@entry=0x55e7e01a3190, subpath=0x7f511867ed68, parallel_aware=parallel_aware@entry=false, ordered=ordered@entry=true,
    nested_oids=0x7f5128aa8698) at /root/timescaledb/src/chunk_append/chunk_append.c:206
206                                     Path *child = (Path *) lfirst(flat);
(gdb) thread apply all bt full

Thread 1 (Thread 0x7f5338d816c0 (LWP 11928)):
#0  ts_chunk_append_path_create (root=root@entry=0x55e7e01ac958, rel=rel@entry=0x55e7e01ab738, ht=ht@entry=0x55e7e01a3190, subpath=0x7f511867ed68, parallel_aware=parallel_aware@entry=false, ordered=ordered@entry=true,
    nested_oids=0x7f5128aa8698) at /root/timescaledb/src/chunk_append/chunk_append.c:206
        child = <optimized out>
        parent_relid = <optimized out>
        is_not_pruned = <optimized out>
        lc_oid = 0x7f5128b23c18
        current_oids = <optimized out>
        merge_childs = 0x0
        append = <optimized out>
        flat = 0x0
        nested_children = 0x7f511867f7d8
        has_scan_childs = false
        path = 0x7f511867ee68
        lc = 0x7f5128b23c88
        rows = 0
        total_cost = 0
        children = <optimized out>
        __func__ = "ts_chunk_append_path_create"
#1  0x00007f512a1407db in apply_optimizations (ht=0x55e7e01a3190, rte=<optimized out>, rel=0x55e7e01ab738, reltype=<optimized out>, root=0x55e7e01ac958) at /root/timescaledb/src/planner.c:691
        pathptr = 0x7f511867edf8
        private = <optimized out>
        ordered = <optimized out>
        order_attno = <optimized out>
        nested_oids = <optimized out>
        lc = 0x7f511867edf8
#2  timescaledb_set_rel_pathlist (rte=<optimized out>, rti=<optimized out>, rel=<optimized out>, root=0x55e7e01ac958) at /root/timescaledb/src/planner.c:775
        reltype = <optimized out>
        ht = 0x55e7e01a3190
        reltype = <optimized out>
        ht = <optimized out>
#3  timescaledb_set_rel_pathlist (root=0x55e7e01ac958, rel=<optimized out>, rti=<optimized out>, rte=<optimized out>) at /root/timescaledb/src/planner.c:734
        reltype = <optimized out>
        ht = <optimized out>
#4  0x000055e7df395dd3 in set_rel_pathlist (root=root@entry=0x55e7e01ac958, rel=0x55e7e01ab738, rti=rti@entry=1, rte=0x55e7e01ab5d8) at ./build/../src/backend/optimizer/path/allpaths.c:540
        __func__ = "set_rel_pathlist"
#5  0x000055e7df396722 in set_base_rel_pathlists (root=0x55e7e01ac958) at ./build/../src/backend/optimizer/path/allpaths.c:352
        rel = <optimized out>
        rti = 1
        rti = <optimized out>
        rel = <optimized out>
#6  make_one_rel (root=root@entry=0x55e7e01ac958, joinlist=joinlist@entry=0x55e7e01c7a28) at ./build/../src/backend/optimizer/path/allpaths.c:222
        rel = <optimized out>
        rti = <optimized out>
        total_pages = 0
#7  0x000055e7df3b691c in query_planner (root=root@entry=0x55e7e01ac958, qp_callback=qp_callback@entry=0x55e7df3b7e60 <standard_qp_callback>, qp_extra=qp_extra@entry=0x7ffe0a31e160) at ./build/../src/backend/optimizer/plan/planmain.c:271
        parse = 0x55e7e01ab4c8
        joinlist = 0x55e7e01c7a28
        final_rel = <optimized out>
        __func__ = "query_planner"
#8  0x000055e7df3bad6a in grouping_planner (root=<optimized out>, inheritance_update=false, tuple_fraction=<optimized out>) at ./build/../src/backend/optimizer/plan/planner.c:2048
        sort_input_targets = 0x55e7e01b8260
        sort_input_target_parallel_safe = <optimized out>
        grouping_target = <optimized out>
        scanjoin_target = <optimized out>
        activeWindows = <optimized out>
        qp_extra = {activeWindows = 0x0, groupClause = 0x55e7e01b8110}
        sort_input_targets_contain_srfs = 0x55e7e00b9c10
        have_grouping = <optimized out>
        wflists = <optimized out>
        gset_data = 0x0
        sort_input_target = <optimized out>
        grouping_targets = 0x10
        grouping_target_parallel_safe = <optimized out>
        scanjoin_targets = 0x55e7e01b8260
        scanjoin_target_parallel_safe = <optimized out>
        grouping_targets_contain_srfs = 0x0
        scanjoin_targets_contain_srfs = 0x55e7e00b9c10
        scanjoin_target_same_exprs = <optimized out>
        agg_costs = {numAggs = 1, numOrderedAggs = 0, hasNonPartial = false, hasNonSerial = false, transCost = {startup = 0, per_tuple = 0.0025000000000000001}, finalCost = {startup = 0, per_tuple = 0}, transitionSpace = 0}
        parse = 0x55e7e01ab4c8
        offset_est = 0
        count_est = 0
        limit_tuples = -1
        have_postponed_srfs = false
        final_target = <optimized out>
        final_targets = 0x18
        final_targets_contain_srfs = 0x0
        final_target_parallel_safe = <optimized out>
        current_rel = <optimized out>
        final_rel = <optimized out>
        extra = {limit_needed = 208, limit_tuples = 4.6666667105823361e-310, count_est = 94454385764536, offset_est = 0}
        lc = <optimized out>
        __func__ = "grouping_planner"
#9  0x000055e7df3bd728 in subquery_planner (glob=glob@entry=0x55e7e01abd88, parse=parse@entry=0x55e7e01ab4c8, parent_root=parent_root@entry=0x0, hasRecursion=hasRecursion@entry=false, tuple_fraction=tuple_fraction@entry=0)
    at ./build/../src/backend/optimizer/plan/planner.c:1012
        root = <optimized out>
        newWithCheckOptions = <optimized out>
        newHaving = <optimized out>
        hasOuterJoins = <optimized out>
        hasResultRTEs = <optimized out>
        final_rel = <optimized out>
        l = <optimized out>
#10 0x000055e7df3be855 in standard_planner (parse=parse@entry=0x55e7e01ab4c8, cursorOptions=cursorOptions@entry=256, boundParams=boundParams@entry=0x0) at ./build/../src/backend/optimizer/plan/planner.c:406
        result = <optimized out>
        glob = 0x55e7e01abd88
        tuple_fraction = 0
        root = <optimized out>
        final_rel = <optimized out>
        best_path = <optimized out>
        top_plan = <optimized out>
        lp = <optimized out>
        lr = <optimized out>
#11 0x00007f512a1401d8 in timescaledb_planner (parse=0x55e7e01ab4c8, cursor_opts=256, bound_params=0x0) at /root/timescaledb/src/planner.c:296
        save_exception_stack = 0x7ffe0a31e7e0
        save_context_stack = 0x0
        local_sigjmp_buf = {{__jmpbuf = {94454385701880, 467523530100873109, 94454385652936, 256, 0, 0, 467523529868089237, 514178261835880341}, __mask_was_saved = 0, __saved_mask = {__val = {94454371544073, 94454384663568, 24, 0,
                94454385652936, 140729069462688, 94454373212138, 94454385701880, 223, 140729069462720, 94454370692649, 94454385701800, 94454385652936, 140729069462752, 94454370692874, 94454385701800}}}}
        stmt = <optimized out>
        lc = <optimized out>
#12 0x000055e7df474642 in pg_plan_query (querytree=querytree@entry=0x55e7e01ab4c8, cursorOptions=cursorOptions@entry=256, boundParams=boundParams@entry=0x0) at ./build/../src/backend/tcop/postgres.c:878
        plan = <optimized out>
#13 0x000055e7df474716 in pg_plan_queries (querytrees=<optimized out>, cursorOptions=cursorOptions@entry=256, boundParams=boundParams@entry=0x0) at ./build/../src/backend/tcop/postgres.c:968
        query = 0x55e7e01ab4c8
        stmt = <optimized out>
        stmt_list = 0x0
        query_list = 0x55e7e01b73f8
#14 0x000055e7df474bcf in exec_simple_query (
    query_string=0x55e7e00b9d20 "SELECT\n    date_trunc('hour', created) AS czas,\n count(*) AS liczba\n  FROM measurement\n  WHERE created BETWEEN '2020-08-10 06:00:00' AND '2020-08-10 10:00:00' OR created BETWEEN '2020-08-11 06:00:00' "...) at ./build/../src/backend/tcop/postgres.c:1143
        parsetree = 0x55e7e00bba80
        portal = <optimized out>
        snapshot_set = true
        commandTag = 0x55e7df6dc764 "SELECT"
        completionTag = "\020\346\061\n\376\177\000\000z\356\062\n\376\177", '\000' <repeats 11 times>, "\006k^\001\000\000\000Q\000\000\000\000\000\000\000\200\347\061\n\376\177\000\000 \235\v\340\347U\000\000؈\021\340\347U\000"
        querytree_list = <optimized out>
        plantree_list = <optimized out>
        receiver = <optimized out>
        format = 0
        dest = DestRemote
        oldcontext = 0x55e7e01a0a80
        parsetree_list = 0x55e7e00bbad0
        parsetree_item = 0x55e7e00bbab0
        save_log_statement_stats = false
        was_logged = false
        use_implicit_block = false
        msec_str = "\020\346\061\n\376\177\000\000z\356\062\n\376\177", '\000' <repeats 11 times>, "\006k^\001\000\000"
        __func__ = "exec_simple_query"
#15 0x000055e7df4764be in PostgresMain (argc=<optimized out>, argv=argv@entry=0x55e7e0118a30, dbname=<optimized out>, username=<optimized out>) at ./build/../src/backend/tcop/postgres.c:4247
        query_string = 0x55e7e00b9d20 "SELECT\n    date_trunc('hour', created) AS czas,\n count(*) AS liczba\n  FROM measurement\n  WHERE created BETWEEN '2020-08-10 06:00:00' AND '2020-08-10 10:00:00' OR created BETWEEN '2020-08-11 06:00:00' "...
        firstchar = <optimized out>
        input_message = {
          data = 0x55e7e00b9d20 "SELECT\n    date_trunc('hour', created) AS czas,\n count(*) AS liczba\n  FROM measurement\n  WHERE created BETWEEN '2020-08-10 06:00:00' AND '2020-08-10 10:00:00' OR created BETWEEN '2020-08-11 06:00:00' "..., len = 303, maxlen = 1024, cursor = 303}
        local_sigjmp_buf = {{__jmpbuf = {140729069463456, 467523530188953493, 1, 94454385051864, 140729069464320, 94454385015728, 467523530019084181, 5931048024258287509}, __mask_was_saved = 1, __saved_mask = {__val = {0, 140729069473485, 140729069473433, 140729069473746, 0, 139637976727552, 94454375833536, 94454384647456, 13141379868869199360, 140729069463888, 94454373320735, 206158430256, 140729069463912, 140729069463712, 13141379868869199360, 94454384639248}}}}
        send_ready_for_query = false
        disable_idle_in_transaction_timeout = false
        __func__ = "PostgresMain"
#16 0x000055e7df3fc208 in BackendRun (port=0x55e7e010fbb0, port=0x55e7e010fbb0) at ./build/../src/backend/postmaster/postmaster.c:4448
        av = 0x55e7e0118a30
        maxac = <optimized out>
        ac = 1
        i = 1
        av = <optimized out>
        maxac = <optimized out>
        ac = <optimized out>
        i = <optimized out>
        __func__ = "BackendRun"
        __errno_location = <optimized out>
        __errno_location = <optimized out>
        __errno_location = <optimized out>
#17 BackendStartup (port=0x55e7e010fbb0) at ./build/../src/backend/postmaster/postmaster.c:4139
        bn = <optimized out>
        pid = <optimized out>
        bn = <optimized out>
        pid = <optimized out>
        __func__ = "BackendStartup"
        __errno_location = <optimized out>
        __errno_location = <optimized out>
        save_errno = <optimized out>
        __errno_location = <optimized out>
        __errno_location = <optimized out>
#18 ServerLoop () at ./build/../src/backend/postmaster/postmaster.c:1704
        port = 0x55e7e010fbb0
        i = <optimized out>
        rmask = {fds_bits = {32, 0 <repeats 15 times>}}
        selres = <optimized out>
        now = <optimized out>
        readmask = {fds_bits = {56, 0 <repeats 15 times>}}
        nSockets = 6
        last_lockfile_recheck_time = 1597152855
        last_touch_time = 1597152794
        __func__ = "ServerLoop"
#19 0x000055e7df3fd12f in PostmasterMain (argc=5, argv=0x55e7e00b3c30) at ./build/../src/backend/postmaster/postmaster.c:1377
        opt = <optimized out>
        status = <optimized out>
        userDoption = <optimized out>
        listen_addr_saved = <optimized out>
        i = <optimized out>
        output_config_variable = <optimized out>
        __func__ = "PostmasterMain"
#20 0x000055e7df166eed in main (argc=5, argv=0x55e7e00b3c30) at ./build/../src/backend/main/main.c:228
No locals.        

Extra info:

Same query without ORDER BY clause gives proper results:

treesat_storage=# SELECT
treesat_storage-#     date_trunc('hour', created) AS czas,
treesat_storage-#     count(*) AS liczba
treesat_storage-#   FROM measurement
treesat_storage-#   WHERE created BETWEEN '2020-08-10 06:00:00' AND '2020-08-10 10:00:00' OR created BETWEEN '2020-08-11 06:00:00' AND '2020-08-11 10:00:00'
treesat_storage-#   GROUP BY date_trunc('hour', created);
          czas          | liczba
------------------------+--------
 2020-08-11 07:00:00+02 |  21348
 2020-08-10 07:00:00+02 |  63874
 2020-08-10 09:00:00+02 |  74849
 2020-08-10 10:00:00+02 |   9269
 2020-08-11 06:00:00+02 |  55830
 2020-08-11 09:00:00+02 |  70210
 2020-08-11 08:00:00+02 |  48567
 2020-08-10 06:00:00+02 |  57930
 2020-08-10 08:00:00+02 |  70739
 2020-08-11 10:00:00+02 |   5489
(10 rows)

And here is total row count for that table:

treesat_storage=# SELECT COUNT(*) FROM measurement ;
   count
-----------
 738107592
(1 row)

svenklemm added a commit to svenklemm/timescaledb that referenced this issue Aug 25, 2020
When postgres prunes children before we create the ChunkAppend
path there might be a mismatch between the children of the path
and the ordered list of children in a space partitioned hypertable.

Fixes timescale#1841
svenklemm added a commit to svenklemm/timescaledb that referenced this issue Aug 25, 2020
When postgres prunes children before we create the ChunkAppend
path there might be a mismatch between the children of the path
and the ordered list of children in a space partitioned hypertable.

Fixes timescale#1841
svenklemm added a commit that referenced this issue Aug 25, 2020
When postgres prunes children before we create the ChunkAppend
path there might be a mismatch between the children of the path
and the ordered list of children in a space partitioned hypertable.

Fixes #1841
svenklemm added a commit that referenced this issue Aug 25, 2020
When postgres prunes children before we create the ChunkAppend
path there might be a mismatch between the children of the path
and the ordered list of children in a space partitioned hypertable.

Fixes #1841
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug segfault Segmentation fault
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants