crashes after executing multiple alter tags (stable reproduction)

[QingZ11]

Background infos:

• version: v2.6.1
• deployment: stand-alone
• installation: RPM
• hardware information
  • Disk 500G
  • CPU 8 cores, memory information 16G

Description:

Under one space, execute multiple query alter tag T1 add (name_xxx string) about 300 time, nebula-graphd and nebula-storaged will crash.

Steps to reproduce the behavior:

1. Step 1: create space S0;
2. Step 2: create tag T0;
3. Step 3: add attributes in batches through code under T0

code

Test
    public void testCreateTag() {      
        for (int i = 0; i < 1; i++) {
            String spaceName = "S" + i;

            for (int j = 0; j < 1; j++) {
                String tagName = "T" + j;
                String createTagStmt = String.format("USE %s; CREATE TAG IF NOT EXISTS %s", spaceName, tagName);
                try {
                    ResultSet resultSet = nebulaSession.execute(createTagStmt);
                    if (!resultSet.isSucceeded()) {
                        log.error("execute: {}, failed: {}", createTagStmt, resultSet.getErrorMessage());
                    } else {
                        log.info("create tag succeed! space: {}, tag: {}", spaceName, j);
                    }
                } catch (Exception e) {
                    log.error("execute nebula sql failed!", e);
                }
                for (int k = 0; k < 300; k++) {
                    String fieldName = tagName + "_" + k;
                    String alterTagStmt = String.format("USE %s; ALTER TAG %s ADD (%s string) ;", spaceName, tagName, fieldName);
                    try {
                        ResultSet resultSet = nebulaSession.execute(alterTagStmt);
                        if (!resultSet.isSucceeded()) {
                            log.error("execute: {}, failed: {}", alterTagStmt, resultSet.getErrorMessage());
                            break;
                        } else {
                            log.info("alter tag succeed! space: {}, tag: {}, field: {}", spaceName, tagName, fieldName);
                        }
                    } catch (Exception e) {
                        log.error("execute nebula sql failed!", e);
                    }
                }
            }
        }
    }

log

*** Aborted at 1644750506 (Unix time, try 'date -d @1644750506') ***
*** Signal 11 (SIGSEGV) (0x0) received by PID 17851 (pthread TID 0x7f11bf1e6700) (linux TID 17936) (code: address not mapped to object), stack trace: ***
/usr/local/nebula/bin/nebula-storaged(_ZN5folly10symbolizer17getStackTraceSafeEPmm+0x31)[0x2132491]
/usr/local/nebula/bin/nebula-storaged(_ZN5folly10symbolizer21SafeStackTracePrinter15printStackTraceEb+0x1b)[0x212a2fb]
/usr/local/nebula/bin/nebula-storaged[0x2128647]
/lib64/libpthread.so.0(+0xf36f)[0x7f11d65ff36f]
/usr/local/nebula/bin/nebula-storaged[0x25a1515]
/usr/local/nebula/bin/nebula-storaged[0x2544b5f]
/usr/local/nebula/bin/nebula-storaged[0x10eab45]
/usr/local/nebula/bin/nebula-storaged(_ZN6nebula4meta20NebulaSchemaProviderD1Ev+0x9b)[0x10eb53b]
/usr/local/nebula/bin/nebula-storaged(_ZNSt10_HashtableIiSt4pairIKiSt6vectorISt10shared_ptrIKN6nebula4meta20NebulaSchemaProviderEESaIS8_EEESaISB_ENSt8__detail10_Select1stESt8equal_toIiESt4hashIiENSD_18_Mod_range_hashingENSD_20_Default_ranged_hashENSD_20_Prime_rehash_policyENSD_17_Hashtable_traitsILb0ELb0ELb1EEEED1Ev+0x87)[0xf7c2c7]
/usr/local/nebula/bin/nebula-storaged(_ZNSt23_Sp_counted_ptr_inplaceIN6nebula4meta14SpaceInfoCacheESaIS2_ELN9__gnu_cxx12_Lock_policyE2EE10_M_disposeEv+0x60)[0x1277610]
/usr/local/nebula/bin/nebula-storaged[0x10ee45b]
/usr/local/nebula/bin/nebula-storaged(_ZNSt10_HashtableIiSt4pairIKiSt10shared_ptrIN6nebula4meta14SpaceInfoCacheEEESaIS7_ENSt8__detail10_Select1stESt8equal_toIiESt4hashIiENS9_18_Mod_range_hashingENS9_20_Default_ranged_hashENS9_20_Prime_rehash_policyENS9_17_Hashtable_traitsILb0ELb0ELb1EEEED1Ev+0x23)[0x122b0a3]
/usr/local/nebula/bin/nebula-storaged(_ZN6nebula4meta10MetaClient8loadDataEv+0x1de2)[0x11b6052]
/usr/local/nebula/bin/nebula-storaged(_ZN6nebula4meta10MetaClient19heartBeatThreadFuncEv+0x227)[0x11b6b67]
/usr/local/nebula/bin/nebula-storaged(_ZZN6nebula6thread13GenericWorker12addDelayTaskIMNS_4meta10MetaClientEFvvEJPS4_EEENSt9enable_ifIXsrSt7is_voidINSt9result_ofIFT_DpT0_EE4typeEE5valueEN5folly10SemiFutureINSI_4UnitEEEE4typeEmOSB_DpOSC_ENKUlvE_clEv+0x1c)[0x12560fc]
/usr/local/nebula/bin/nebula-storaged[0x18416aa]
/usr/local/nebula/bin/nebula-storaged[0x219bd92]
/usr/local/nebula/bin/nebula-storaged(event_base_loop+0x506)[0x219c466]
/usr/local/nebula/bin/nebula-storaged(_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJPFvRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt8functionIFvvEEES8_St5_BindIFMN6nebula6thread13GenericWorkerEFvvEPSL_EEEEEEE6_M_runEv+0x5c)[0x1841c2c]
/usr/local/nebula/bin/nebula-storaged[0x266025f]
/lib64/libpthread.so.0(+0x7dc4)[0x7f11d65f7dc4]
/lib64/libc.so.6(clone+0x6c)[0x7f11d632676c]
(safe mode, symbolizer not available)

ref: https://discuss.nebula-graph.com.cn/t/topic/7522

[sworduo]

I find another question related to this problem. If the version of schema is greater than 256, let's say 500, then 'desc tag xxx' would only return the schema with version 255 but not the largest one.

The resolution may be iterated all the versions of this tag/edge to find the largest one in GetTagProcessor, I'm not sure.

[HarrisChu]

good catch! could you raise a new issue?

in little littleEndian

1<<63-1 => [255 255 255 255 255 255 255 127]
1<<63-2 => [254 255 255 255 255 255 255 127]
1<<63-1-255 => [0 255 255 255 255 255 255 127]
1<<63-1-256 => [255 254 255 255 255 255 255 127]

when we doPrefix tag schema, would get ver 255.
there's the same issue for edge schema.