Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cosbench read test is terminated #357

Open
akumacxd opened this issue Aug 12, 2017 · 7 comments
Open

cosbench read test is terminated #357

akumacxd opened this issue Aug 12, 2017 · 7 comments

Comments

@akumacxd
Copy link

akumacxd commented Aug 12, 2017 •
edited

Hi

Why is the reading test terminated ? Here is my environment details
COSBench 0.4.2 （SUSE12SP3）
Ceph accessing via S3

ceph -s
cluster:
id: 1d803222-224e-3c69-a20e-3fd45579a55b
health: HEALTH_OK

services:
mon: 3 daemons, quorum node001,node002,node003
mgr: node001(active), standbys: node002, node003
mds: 1/1/1 up {0=node001=up:active}
osd: 40 osds: 40 up, 40 in

data:
pools: 14 pools, 886 pgs
objects: 168k objects, 312 GB
usage: 708 GB used, 144 TB / 145 TB avail
pgs: 886 active+clean

io:
client: 85 B/s rd, 0 B/s wr, 0 op/s rd, 0 op/s wr

ceph -v
ceph version 12.1.0-671-g2c11b88d14 (2c11b88d14e64bf60c0556c6a4ec8c9eda36ff6a) luminous (rc)

the system logs:

2017-08-12 21:29:47,674 [ERROR] [AbstractAgent] - unexpected exception
java.lang.ArrayIndexOutOfBoundsException: 0
at com.intel.cosbench.driver.operator.AbstractOperator.errorStatisticsHandle(AbstractOperator.java:117)
at com.intel.cosbench.driver.operator.Reader.doRead(Reader.java:103)
at com.intel.cosbench.driver.operator.Reader.operate(Reader.java:69)
at com.intel.cosbench.driver.operator.AbstractOperator.operate(AbstractOperator.java:76)
at com.intel.cosbench.driver.agent.WorkAgent.performOperation(WorkAgent.java:197)
at com.intel.cosbench.driver.agent.WorkAgent.doWork(WorkAgent.java:177)
at com.intel.cosbench.driver.agent.WorkAgent.execute(WorkAgent.java:134)
at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:44)
at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:1)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
2017-08-12 21:29:47,677 [ERROR] [AbstractAgent] - unexpected exception
java.lang.ArrayIndexOutOfBoundsException: 0
at com.intel.cosbench.driver.operator.AbstractOperator.errorStatisticsHandle(AbstractOperator.java:117)
at com.intel.cosbench.driver.operator.Reader.doRead(Reader.java:103)
at com.intel.cosbench.driver.operator.Reader.operate(Reader.java:69)
at com.intel.cosbench.driver.operator.AbstractOperator.operate(AbstractOperator.java:76)
at com.intel.cosbench.driver.agent.WorkAgent.performOperation(WorkAgent.java:197)
at com.intel.cosbench.driver.agent.WorkAgent.doWork(WorkAgent.java:177)
at com.intel.cosbench.driver.agent.WorkAgent.execute(WorkAgent.java:134)
at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:44)
at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:1)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
2017-08-12 21:29:47,677 [ERROR] [AbstractAgent] - unexpected exception
java.lang.ArrayIndexOutOfBoundsException: 0
at com.intel.cosbench.driver.operator.AbstractOperator.errorStatisticsHandle(AbstractOperator.java:117)
at com.intel.cosbench.driver.operator.Reader.doRead(Reader.java:103)
at com.intel.cosbench.driver.operator.Reader.operate(Reader.java:69)
at com.intel.cosbench.driver.operator.AbstractOperator.operate(AbstractOperator.java:76)
at com.intel.cosbench.driver.agent.WorkAgent.performOperation(WorkAgent.java:197)
at com.intel.cosbench.driver.agent.WorkAgent.doWork(WorkAgent.java:177)
at com.intel.cosbench.driver.agent.WorkAgent.execute(WorkAgent.java:134)
at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:44)
at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:1)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

2017-08-12 21:33:05,455 [ERROR] [MissionHandler] - detected workers [25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 51, 52, 53, 54, 55, 58] have encountered errors
2017-08-12 21:33:05,461 [INFO] [MissionHandler] - mission M2D6A10D362 has been terminated
2017-08-12 21:33:05,778 [INFO] [StageRunner] - successfully queried all tasks in stage s6-main
2017-08-12 21:33:05,781 [ERROR] [AbstractCommandTasklet] - driver report error: HTTP 409 - no mission handler
2017-08-12 21:33:05,781 [ERROR] [StageRunner] - detected tasks [t1] have encountered errors
2017-08-12 21:33:05,781 [INFO] [StageRunner] - begin to terminate stage s6-main
2017-08-12 21:33:05,892 [INFO] [StageRunner] - stage s6-main has been terminated
2017-08-12 21:33:08,146 [ERROR] [WorkloadProcessor] - detected stage s6-main encountered error
2017-08-12 21:33:08,146 [INFO] [WorkloadProcessor] - begin to terminate workload w130
2017-08-12 21:33:08,225 [INFO] [SimpleWorkloadArchiver] - workload w130 has been successfully archived

===============Here is my controller config file
cat conf/controller.conf
[controller]
concurrency=1
drivers=1
log_level = INFO
log_file = log/system.log
archive_dir = archive

[driver1]
name = driver1
url = http://10.223.30.174:18088/driver

===============Here is my workload config file

    <workstage name="init">
      <work type="init" workers="8" config="cprefix=s3testsuse;containers=r(1,100)" />
    </workstage>

    <workstage name="prepare">
      <work type="prepare" workers="8" config="cprefix=s3testsuse;containers=r(1,100);objects=r(1,100);sizes=c(100)KB" />
    </workstage>
	
	<workstage name="delay" closuredelay="30">
	  <work type="delay" workers="1" />
	</workstage>

	<workstage name="main">
      <work name="main" type="normal" workers="200" runtime="300">
       <!--   <operation type="read" ratio="100" config="cprefix=s3testsuse;containers=u(1,2);objects=u(1,10)" />-->
          <operation type="write" ratio="100" config="cprefix=s3testsuse;containers=u(1,100);objects=u(1,100);sizes=c(100)KB" />  
      </work>
    </workstage>
	
	<workstage name="delay" closuredelay="30">
	  <work type="delay" workers="1" />
	</workstage>
	
    <workstage name="main">
      <work name="main" type="normal" workers="130" runtime="300" rampup="20" afr="200000">
       <!--   <operation type="read" ratio="100" config="cprefix=s3testsuse;containers=u(1,2);objects=u(1,10)" />-->
	      <operation type="read" ratio="100" config="cprefix=s3testsuse;containers=u(1,100);objects=u(1,100)" /> 
      </work>
    </workstage>

	<workstage name="cleanup">
      <work type="cleanup" workers="8" config="cprefix=s3testsuse;containers=r(1,100);objects=r(1,100)" />
    </workstage>

    <workstage name="dispose">
      <work type="dispose" workers="8" config="cprefix=s3testsuse;containers=r(1,100)" />
    </workstage>
@akumacxd
Copy link
Author

image 1

@akumacxd
Copy link
Author

s3cmd ls
2017-08-12 13:20 s3://s3testsuse1
2017-08-12 13:20 s3://s3testsuse10
2017-08-12 13:20 s3://s3testsuse100
2017-08-12 13:20 s3://s3testsuse11
2017-08-12 13:20 s3://s3testsuse12
2017-08-12 13:20 s3://s3testsuse13
2017-08-12 13:20 s3://s3testsuse14
2017-08-12 13:20 s3://s3testsuse15

@qiuyi21
Copy link

qiuyi21 commented Oct 8, 2017

Me too

@prtsh
Copy link

prtsh commented Dec 21, 2017 •
edited

Same here -

Here is the config for which it's getting terminated -

This config is for Ceph with RGW S3

<workstage name="init">
  <work type="init" workers="1" config="cprefix=s3testqwer;containers=r(1,2)" />
</workstage>

<workstage name="prepare">
  <work type="prepare" workers="1" config="cprefix=s3testqwer;containers=r(1,2);objects=r(1,10);sizes=c(64)KB" />
</workstage>

<workstage name="main">
  <work name="main" workers="64" runtime="30">
    <operation type="read" ratio="100" config="cprefix=s3testqwer;containers=u(1,2);objects=u(1,10)" />
    <!--operation type="write" ratio="0" config="cprefix=s3testqwer;containers=u(1,2);objects=u(11,20);sizes=c(4)MB" -->
  </work>
</workstage>

<workstage name="cleanup">
  <work type="cleanup" workers="1" config="cprefix=s3testqwer;containers=r(1,2);objects=r(1,20)" />
</workstage>

<workstage name="dispose">
  <work type="dispose" workers="1" config="cprefix=s3testqwer;containers=r(1,2)" />
</workstage>

@style1999
Copy link

me, too. if the main workstage includes write operations only, it will complete smoothly; but the job will be terminated if the conf-file includes read operations in the main workstage.

@rongzhen-zhan
Copy link

We also encountered the same erro.How to solve it

@haoxihuanni
Copy link

How to solve it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@prtsh @qiuyi21 @akumacxd @rongzhen-zhan @haoxihuanni @style1999