We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
配置内容:
"job": { "setting": { "speed": { "channel": **5,** //"record": 200, "bytes": -1
错误信息:
2021-02-22 08:21:00.688 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0 2021-02-22 08:21:00.688 [main] INFO JobContainer - DataX jobContainer starts job. 2021-02-22 08:21:00.691 [main] INFO JobContainer - Set jobId = 0 2021-02-22 08:21:00.711 [job-0] INFO JobContainer - DataX Reader.Job [streamreader] do prepare work . 2021-02-22 08:21:00.711 [job-0] INFO JobContainer - DataX Writer.Job [influxdbwriter] do prepare work . 2021-02-22 08:21:10.613 [job-0] INFO JobContainer - Job set Channel-Number to 3 channels. 2021-02-22 08:21:10.614 [job-0] INFO JobContainer - DataX Reader.Job [streamreader] splits to [3] tasks. 2021-02-22 08:21:10.615 [job-0] WARN InfluxDBWriter$Job - -------------------------split() begin... 2021-02-22 08:21:10.615 [job-0] INFO JobContainer - DataX Writer.Job [influxdbwriter] splits to [1] tasks. 2021-02-22 08:21:10.620 [job-0] INFO StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00% 2021-02-22 08:21:10.621 [job-0] ERROR Engine - Code:[Framework-15], Description:[DataX插件切分出错, 该问题通常是由于DataX各个插件编程错误引起,请联系DataX开发团队解决]. - Code:[Framework-15], Description:[DataX插件切分出错, 该问题通常是由于DataX各个插件编程错误引起,请联系DataX开发团队解决]. - reader切分的task数目[3]不等于writer切分的task数目[1].
代码修改: // @OverRide // public List split(int adviceNumber) // { // Configuration readerSliceConfig = super.getPluginJobConf(); // List splittedConfigs = new ArrayList<>(); // splittedConfigs.add(readerSliceConfig); // return splittedConfigs;
// } @Override public List<Configuration> split(int mandatoryNumber) { //修改多channel无效问题 LOG.warn("-------------------------split [{}] begin...", mandatoryNumber); ArrayList<Configuration> configurations = new ArrayList<Configuration>(mandatoryNumber); for (int i = 0; i < mandatoryNumber; i++) { configurations.add(this.originalConfig.clone()); } return configurations; }
初步测试设置多channel后无报错。
The text was updated successfully, but these errors were encountered:
Add support for more than one channel setup (#130)
46a789f
Fixed #129
感谢反馈并提供修复代码,已经修复
Sorry, something went wrong.
7fa4ebb
Successfully merging a pull request may close this issue.
配置内容:
错误信息:
2021-02-22 08:21:00.688 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2021-02-22 08:21:00.688 [main] INFO JobContainer - DataX jobContainer starts job.
2021-02-22 08:21:00.691 [main] INFO JobContainer - Set jobId = 0
2021-02-22 08:21:00.711 [job-0] INFO JobContainer - DataX Reader.Job [streamreader] do prepare work .
2021-02-22 08:21:00.711 [job-0] INFO JobContainer - DataX Writer.Job [influxdbwriter] do prepare work .
2021-02-22 08:21:10.613 [job-0] INFO JobContainer - Job set Channel-Number to 3 channels.
2021-02-22 08:21:10.614 [job-0] INFO JobContainer - DataX Reader.Job [streamreader] splits to [3] tasks.
2021-02-22 08:21:10.615 [job-0] WARN InfluxDBWriter$Job - -------------------------split() begin...
2021-02-22 08:21:10.615 [job-0] INFO JobContainer - DataX Writer.Job [influxdbwriter] splits to [1] tasks.
2021-02-22 08:21:10.620 [job-0] INFO StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00%
2021-02-22 08:21:10.621 [job-0] ERROR Engine - Code:[Framework-15], Description:[DataX插件切分出错, 该问题通常是由于DataX各个插件编程错误引起,请联系DataX开发团队解决]. - Code:[Framework-15], Description:[DataX插件切分出错, 该问题通常是由于DataX各个插件编程错误引起,请联系DataX开发团队解决]. - reader切分的task数目[3]不等于writer切分的task数目[1].
代码修改:
// @OverRide
// public List split(int adviceNumber)
// {
// Configuration readerSliceConfig = super.getPluginJobConf();
// List splittedConfigs = new ArrayList<>();
// splittedConfigs.add(readerSliceConfig);
// return splittedConfigs;
初步测试设置多channel后无报错。
The text was updated successfully, but these errors were encountered: