how to increase the writing throughput

[rainaftersunshine]

场景:
1.版本 client version:1.6.2.0 server version:1.6.2.0
1.CREATE DATABASE tdb precision 'us';
2.创建 1张10列左右的超级表, 1张 30列左右的超级表
3.采用异步接口插入数据
taos_query_a(m_pTaos, (char *)sSql.c_str(), InsertSnapshotDataCallBack, pParam);
4.插入语句 insert into xxx using xxx tag xxx
5.数据峰值3w-6w/s, 平时6k-8k/s 笔记录

尝试步骤:
1.采用单线程单个m_pTaos, 单记录insert into 出现性能问题,记录无法插入太慢。
2.采用多线程单个m_pTaos, 单记录insert into, 出现插入失败-65, TSDB_CODE_MAX_SESSIONS
3.采用多线程多个m_pTaos,单记录insert into, 出现插入失败-65, TSDB_CODE_MAX_SESSIONS
4.采用单线程单个m_pTaos, 多记录批量insert, 单次批量插入的记录超过50笔时出现 -1， TSDB_CODE_ACTION_IN_PROGRESS
5.目前采用单线程单个m_pTaos, 多记录批量insert, 单次批量插入的记录20笔, 部分解决性能问题，但是数据峰值的时候会出现大量数据挤压，延迟5分左右才能全部处理完成。

针对这种情况， 有什么方案能提高写入速度呢，也尝试修改过服务器配置中的相关sessions最大值均无效。

[jtao1735]

May you post your inserting data records code here? We can take a look and find out the bottleneck. In the testing, single core can handle at least 20K insert requests per second. There must be something not efficient in your code.

For TSDB_CODE_MAX_SESSIONS error, please set debugFlag to 135 in taos.cfg in both client and server, and re-run the testing, and post the log file there. We can tell why

[rainaftersunshine]

===============================tasod info============================
9/27 08:54:54.198531 7f00b22a1700 UTL locale not configured, set to system default:en_US.UTF-8
09/27 08:54:54.198554 7f00b22a1700 UTL charset not configured, set to system default:UTF-8
09/27 08:54:54.198665 7f00b22a1700 UTL taos config & system info:
09/27 08:54:54.198694 7f00b22a1700 UTL ==================================
09/27 08:54:54.198703 7f00b22a1700 UTL internalIp: 198.28.2.65
09/27 08:54:54.198711 7f00b22a1700 UTL serverIp: 0.0.0.0
09/27 08:54:54.198717 7f00b22a1700 UTL localIp: 0.0.0.0
09/27 08:54:54.198724 7f00b22a1700 UTL httpIp: 0.0.0.0
09/27 08:54:54.198730 7f00b22a1700 UTL httpPort: 6020
09/27 08:54:54.198740 7f00b22a1700 UTL mgmtShellPort: 6030
09/27 08:54:54.198747 7f00b22a1700 UTL vnodeShellPort: 6035
09/27 08:54:54.198757 7f00b22a1700 UTL configDir: /etc/taos
09/27 08:54:54.198764 7f00b22a1700 UTL dataDir: /data/taos/datadir
09/27 08:54:54.198771 7f00b22a1700 UTL logDir: /data/taos/logdir
09/27 08:54:54.198779 7f00b22a1700 UTL scriptDir: /etc/taos
09/27 08:54:54.198786 7f00b22a1700 UTL numOfThreadsPerCore: 1.000000
09/27 08:54:54.198818 7f00b22a1700 UTL ratioOfQueryThreads: 0.500000
09/27 08:54:54.198829 7f00b22a1700 UTL numOfVnodesPerCore: 8
09/27 08:54:54.198836 7f00b22a1700 UTL numOfTotalVnodes: 0
09/27 08:54:54.198842 7f00b22a1700 UTL checkHeaderFile: 0
09/27 08:54:54.198853 7f00b22a1700 UTL tables: 1000
09/27 08:54:54.198868 7f00b22a1700 UTL cache: 16384(byte)
09/27 08:54:54.198876 7f00b22a1700 UTL rows: 4096
09/27 08:54:54.198884 7f00b22a1700 UTL fileBlockMinPercent: 0.050000
09/27 08:54:54.198892 7f00b22a1700 UTL ablocks: 4
09/27 08:54:54.198899 7f00b22a1700 UTL tblocks: 100
09/27 08:54:54.198907 7f00b22a1700 UTL monitorInterval: 30(s)
09/27 08:54:54.198922 7f00b22a1700 UTL rpcTimer: 300(ms)
09/27 08:54:54.198930 7f00b22a1700 UTL rpcMaxTime: 600(s)
09/27 08:54:54.198938 7f00b22a1700 UTL ctime: 3600(s)
09/27 08:54:54.198945 7f00b22a1700 UTL statusInterval: 1(s)
09/27 08:54:54.198951 7f00b22a1700 UTL shellActivityTimer: 3(s)
09/27 08:54:54.198959 7f00b22a1700 UTL meterMetaKeepTimer: 7200(s)
09/27 08:54:54.198966 7f00b22a1700 UTL metricMetaKeepTimer: 600(s)
09/27 08:54:54.198973 7f00b22a1700 UTL maxUsers: 1000
09/27 08:54:54.198992 7f00b22a1700 UTL maxDbs: 1000
09/27 08:54:54.198999 7f00b22a1700 UTL maxTables: 650000
09/27 08:54:54.199006 7f00b22a1700 UTL maxVGroups: 1000
09/27 08:54:54.199015 7f00b22a1700 UTL minSlidingTime: 10(ms)
09/27 08:54:54.199023 7f00b22a1700 UTL minIntervalTime: 10(ms)
09/27 08:54:54.199032 7f00b22a1700 UTL maxStreamCompDelay: 20000(ms)
09/27 08:54:54.199040 7f00b22a1700 UTL maxFirstStreamCompDelay:10000(ms)
09/27 08:54:54.199046 7f00b22a1700 UTL retryStreamCompDelay: 10(ms)
09/27 08:54:54.199055 7f00b22a1700 UTL clog: 1
09/27 08:54:54.199062 7f00b22a1700 UTL comp: 2
09/27 08:54:54.199071 7f00b22a1700 UTL days: 10
09/27 08:54:54.199077 7f00b22a1700 UTL keep: 3650
09/27 08:54:54.199084 7f00b22a1700 UTL defaultDB:
09/27 08:54:54.199092 7f00b22a1700 UTL defaultUser: root
09/27 08:54:54.199098 7f00b22a1700 UTL timezone: (CST, +0800)
09/27 08:54:54.199107 7f00b22a1700 UTL locale: en_US.UTF-8
09/27 08:54:54.199113 7f00b22a1700 UTL charset: UTF-8
09/27 08:54:54.199121 7f00b22a1700 UTL maxShellConns: 20000
09/27 08:54:54.199128 7f00b22a1700 UTL maxMeterConnections: 100000
09/27 08:54:54.199134 7f00b22a1700 UTL maxMgmtConnections: 20000
09/27 08:54:54.199142 7f00b22a1700 UTL maxVnodeConnections: 100000
09/27 08:54:54.199149 7f00b22a1700 UTL minimalLogDirGB: 0.100000(GB)
09/27 08:54:54.199213 7f00b22a1700 UTL minimalTmpDirGB: 0.100000(GB)
09/27 08:54:54.199223 7f00b22a1700 UTL minimalDataDirGB: 0.500000(GB)
09/27 08:54:54.199231 7f00b22a1700 UTL enableHttp: 1
09/27 08:54:54.199238 7f00b22a1700 UTL enableMonitor: 1
09/27 08:54:54.199244 7f00b22a1700 UTL httpCacheSessions: 100
09/27 08:54:54.199251 7f00b22a1700 UTL telegrafUseFieldNum: 0
09/27 08:54:54.199257 7f00b22a1700 UTL httpMaxThreads: 2
09/27 08:54:54.199264 7f00b22a1700 UTL restfulRowLimit: 10240
09/27 08:54:54.199270 7f00b22a1700 UTL httpEnableCompress: 0
09/27 08:54:54.199276 7f00b22a1700 UTL numOfLogLines: 10000000
09/27 08:54:54.199283 7f00b22a1700 UTL asyncLog: 1
09/27 08:54:54.199289 7f00b22a1700 UTL debugFlag: 131
09/27 08:54:54.199295 7f00b22a1700 UTL mDebugFlag: 135
09/27 08:54:54.199301 7f00b22a1700 UTL dDebugFlag: 131
09/27 08:54:54.199313 7f00b22a1700 UTL sdbDebugFlag: 135
09/27 08:54:54.199320 7f00b22a1700 UTL taosDebugFlag: 131
09/27 08:54:54.199329 7f00b22a1700 UTL tmrDebugFlag: 131
09/27 08:54:54.199336 7f00b22a1700 UTL cDebugFlag: 131
09/27 08:54:54.199342 7f00b22a1700 UTL jniDebugFlag: 131
09/27 08:54:54.199349 7f00b22a1700 UTL odbcDebugFlag: 131
09/27 08:54:54.199355 7f00b22a1700 UTL uDebugFlag: 131
09/27 08:54:54.199361 7f00b22a1700 UTL httpDebugFlag: 131
09/27 08:54:54.199368 7f00b22a1700 UTL monitorDebugFlag: 131
09/27 08:54:54.199374 7f00b22a1700 UTL qDebugFlag: 131
09/27 08:54:54.199380 7f00b22a1700 UTL gitinfo: b675c55
09/27 08:54:54.199388 7f00b22a1700 UTL buildinfo: Built by ubuntu at 2019-09-12 22:45
09/27 08:54:54.199395 7f00b22a1700 UTL version: 1.6.2.0
09/27 08:54:54.199402 7f00b22a1700 UTL dataDir: /data/taos/datadir
09/27 08:54:54.199408 7f00b22a1700 UTL os pageSize: 4096(KB)
09/27 08:54:54.199414 7f00b22a1700 UTL os openMax: 1024
09/27 08:54:54.199429 7f00b22a1700 UTL os streamMax: 16
09/27 08:54:54.199438 7f00b22a1700 UTL os numOfCores: 4
09/27 08:54:54.199445 7f00b22a1700 UTL os totalDisk: 11.490520(GB)
09/27 08:54:54.199454 7f00b22a1700 UTL os totalMemory: 8001(MB)
09/27 08:54:54.199466 7f00b22a1700 UTL os sysname: Linux
09/27 08:54:54.199476 7f00b22a1700 UTL os nodename: RHEL6.9
09/27 08:54:54.199483 7f00b22a1700 UTL os release: 2.6.32-696.el6.x86_64
09/27 08:54:54.199490 7f00b22a1700 UTL os version: #1 SMP Tue Feb 21 00:53:17 EST 2017
09/27 08:54:54.199497 7f00b22a1700 UTL os machine: x86_64
09/27 08:54:54.199503 7f00b22a1700 UTL ==================================

inserting code:
void TdbMgr::WorkBatchInsert()
{
m_Lock.Acquire();
nt nTotal = 0;
int nInsertedRow = 0;
if (m_mapWaitInsertSnap.size() > 0)
{
string sSql = "insert into ";
map<string, vector<SnapShot *> *>::iterator it;

    for(it = m_mapWaitInsertSnap.begin(); it != m_mapWaitInsertSnap.end(); it++)
    {
        nTotal = nTotal + it->second->size();
    }
    for(it = m_mapWaitInsertSnap.begin(); it != m_mapWaitInsertSnap.end();)
    {
        //加上本次最多插入50条记录
        if (nInsertedRow + it->second->size() > m_nPerMaxInsert)
        {
            break;
        }
        
        sSql = sSql + GenerateInsertSql(it->second, it->first);
        
       nInsertedRow += it->second->size();

       if (it->second != NULL)
        {
            for(int i = 0; i < it->second->size(); i++)
            {
                if (it->second->at(i) != NULL)
                    delete it->second->at(i);
            }
            delete it->second;
        }
        m_mapWaitInsertSnap.erase(it++);
    }
    sSql = sSql + ";";
    TaosParam * pParam = new TaosParam();
    pParam->nRow = nInsertedRow;
    taos_query_a(m_pTaos, (char *)sSql.c_str(), InsertSnapshotDataCallBack, pParam);
}
m_Lock.Release();

}
void TdbMgr::BatchInsertTranscationData(vector<TickTranscation *> * pVectors, string sMarket, int nMarketDate)
{
if (pVectors != NULL && pVectors->size() > 0)
{
string sSql = "";
char pInsert[256] = "";
string sStockCode = pVectors->at(0)->sStockCode;
sprintf(pInsert, "insert into %s.t%s using %s.transcation tags ('%s') (ts, stock_code, market, market_date, srv_timestamp, price, amount, trade_count, dir)
values ", m_szDbName, sStockCode.c_str(), m_szDbName, sStockCode.c_str());
sSql = sSql + pInsert;
for(int i = 0; i < pVectors->size(); i++)
{
TickTranscation * tick = pVectors->at(i);
char szValue[256] = "";
sprintf(szValue, " (now, '%s', '%s', %d, %d, %d, %d, %d, %d)", tick->sStockCode.c_str(), tick->sMarket.c_str(), tick->nMarketDate,
tick->nTimeStamp, tick->nPrice, tick->lAmount, tick->nCount, tick->nDir);
//printf("szValue=%s\n", szValue);
sSql = sSql + szValue;
}
sSql = sSql + ";";
TaosParam * pParam = new TaosParam();
pParam->nRow = pVectors->size();
taos_query_a(m_pTaos, (char *)sSql.c_str(), InsertTranscationDataCallBack, pParam);
}
}

void TdbMgr::BatchInsertTranscationData(vector<TickTranscation *> * pVectors, string sMarket, int nMarketDate)
{
if (pVectors != NULL && pVectors->size() > 0)
{
string sSql = "";
char pInsert[256] = "";
string sStockCode = pVectors->at(0)->sStockCode;
sprintf(pInsert, "insert into %s.t%s using %s.transcation tags ('%s') (ts, stock_code, market, market_date, srv_timestamp, price, amount, trade_count, dir)
values ", m_szDbName, sStockCode.c_str(), m_szDbName, sStockCode.c_str());
sSql = sSql + pInsert;
for(int i = 0; i < pVectors->size(); i++)
{
TickTranscation * tick = pVectors->at(i);
char szValue[256] = "";
sprintf(szValue, " (now, '%s', '%s', %d, %d, %d, %d, %d, %d)", tick->sStockCode.c_str(), tick->sMarket.c_str(), tick->nMarketDate,
tick->nTimeStamp, tick->nPrice, tick->lAmount, tick->nCount, tick->nDir);
//printf("szValue=%s\n", szValue);
sSql = sSql + szValue;
}
sSql = sSql + ";";
TaosParam * pParam = new TaosParam();
pParam->nRow = pVectors->size();
taos_query_a(m_pTaos, (char *)sSql.c_str(), InsertTranscationDataCallBack, pParam);
}
}

[rainaftersunshine]

我测试环境使用4线程调用BatchInsertTranscationData插入数据
出现新报错信息: tbd插入逐笔数据失败:-29
日志.zip

[rainaftersunshine]

我生产环境使用单线程调用BatchInsertTranscationData插入数据
出现新报错信息: tbd插入逐笔数据失败:-65

taosdlog.0：
taosdlog.0.zip
客户端日志太大无法上传， 我截取部分报错时间点的日志
taoslog0.0:
09/27 09:18:26.707929 7f2c0a803700 TSC 0x7f2bbbeb1640 Async SQL result:-65 taosres:(nil)
09/27 09:18:26.707933 7f2c0a803700 TSC 0x7f2bbbeb1640 object should be release since all data blocks have been submit
09/27 09:18:26.707935 7f2c0a803700 TSC 0x7f2bbbeb1640 Async insertion completed, destroy data block list
09/27 09:18:26.707958 7f2c00f8f700 RPC TSC-mgmt cid:0 sid:1417 id:root, meter-info-rsp received from 0x100007f:6030, parse code:0, first:0 len:4389 tranId:16320 pConn:0x7f2c001967f0
09/27 09:18:26.707967 7f2c00f8f700 RPC TSC-mgmt cid:0 sid:1417 id:root, meter-info-rsp is put into queue, msgLen:4357 pConn:0x7f2c001967f0 pTimer:(nil)
09/27 09:18:26.709110 7f2c0b204700 RPC TSC-mgmt msg is sent to 0x100007f:6030 len:933 ret:933 localPort:56411 chandle:0x22c61f8
09/27 09:18:26.709124 7f2c0b204700 RPC TSC-mgmt msg sent, len:933 source:0xbf050000 dest:0x80140000 tranId:46528 pConn:0x7f2c00199730
09/27 09:18:26.709131 7f2c0b204700 TSC 0x7f2bf021c0b0 send msg ret:865 code:0 sig:0x7f2bf021c0b0
09/27 09:18:26.709139 7f2c0b204700 TSC 0x7f2bd8e2fad0 start to free sql object
09/27 09:18:26.709145 7f2c0b204700 TSC 0x7f2bd8e2fad0 start to free local reducer
09/27 09:18:26.709148 7f2c0b204700 TSC 0x7f2bd8e2fad0 local reducer has been freed, abort
09/27 09:18:26.709152 7f2c0b204700 UTL key:0.tdb.s143457 is released by app.refcnt:0
09/27 09:18:26.709157 7f2c0b204700 TSC 0x7f2bd8e2fad0 Async sql is automatically freed
09/27 09:18:26.709161 7f2c0b204700 RPC TSC-mgmt cid:0 sid:1418 id:root, meter-info is sent to 127.0.0.1:6030, len:933 tranId:52647 pConn:0x7f2c001968d0
09/27 09:18:26.707977 7f2c00f8f700 RPC TSC-mgmt msg is recv from 0x100007f:6030 len:4389
09/27 09:18:26.709246 7f2c00f8f700 RPC TSC-mgmt msg received, len:4389 source:0x80140000 dest:0xbf050000 tranId:46528
09/27 09:18:26.702970 7f2bfdce7700 UTL key:0.tdb.transcation is retrieved from cache,refcnt:1
09/27 09:18:26.709277 7f2bfdce7700 TSC 0x7f2be8198b70 the number of columns:9, numOfTags:1, addr:0x7f2bd897c19c
09/27 09:18:26.709290 7f2bfdce7700 UTL key:0.tdb.transcation is released by app.refcnt:0
09/27 09:18:26.709296 7f2bfdce7700 UTL key:0.tdb.t600996 is retrieved from cache,refcnt:21
09/27 09:18:26.709300 7f2bfdce7700 TSC 0x7f2be8198b70 the number of columns:9, numOfTags:1, addr:0x7f2bf0f77d4c
09/27 09:18:26.709326 7f2c00f8f700 RPC TSC-mgmt cid:0 sid:1471 id:root, meter-info-rsp received from 0x100007f:6030, parse code:0, first:0 len:4389 tranId:46528 pConn:0x7f2c00199730
09/27 09:18:26.709330 7f2bfdce7700 TSC 0x7f2be8198b70 meterId:0.tdb.t600996, sid:565, rows:1, sversion:0
09/27 09:18:26.709332 7f2c00f8f700 RPC TSC-mgmt cid:0 sid:1471 id:root, meter-info-rsp is put into queue, msgLen:4357 pConn:0x7f2c00199730 pTimer:(nil)
09/27 09:18:26.709346 7f2bfdce7700 UTL key:0.tdb.t600996 is released by app.refcnt:20
09/27 09:18:26.709393 7f2bfdce7700 UTL key:0.tdb.t600996 is retrieved from cache,refcnt:21
09/27 09:18:26.709401 7f2bfdce7700 TSC 0x7f2be8198b70 the number of columns:9, numOfTags:1, addr:0x7f2bf0f77d4c
09/27 09:18:26.709407 7f2bfdce7700 TSC 0x7f2be8198b70 SQL cmd:2 will be processed, name:0.tdb.t600996
09/27 09:18:26.709412 7f2bfdce7700 TSC 0x7f2be8198b70 update submit msg vnode:23
09/27 09:18:26.709437 7f2c0b204700 RPC TSC-mgmt msg is sent to 0x100007f:6030 len:933 ret:933 localPort:56411 chandle:0x22c61f8
09/27 09:18:26.709445 7f2c0b204700 RPC TSC-mgmt msg sent, len:933 source:0x8a050000 dest:0x81140000 tranId:52647 pConn:0x7f2c001968d0
09/27 09:18:26.709452 7f2c0b204700 TSC 0x7f2bbbea0b60 msg:0x7f2bc002be8c is received from server, pConn:0x7f2c017799b0
09/27 09:18:26.709458 7f2c0b204700 TSC 0x7f2bbbea0b60 cmd:2 code:65, inserted rows:0, rsp len:1
09/27 09:18:26.709641 7f2c0b204700 TSC 0x7f2c017799b0 ip:0x0:24:476:0x7f2bff511f90 added, connections in cache:2
09/27 09:18:26.709648 7f2c0b204700 TSC 0x7f2bbbea0b60 Async SQL result:-65 taosres:(nil)

[sangshuduo]

请问还有问题吗？我们已经发布了2.0.3.0版本，请使用最新的版本测试。