Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[QUESTION]About the commit record for issues#79 #1945

Closed
zhaoyw1 opened this issue Feb 12, 2020 · 9 comments
Closed

[QUESTION]About the commit record for issues#79 #1945

zhaoyw1 opened this issue Feb 12, 2020 · 9 comments
Labels
question Further information is requested

Comments

@zhaoyw1
Copy link

zhaoyw1 commented Feb 12, 2020

*For better global communication, please give priority to using English description, thx! *

Describe the question
Recently our team wanna subimit spark task by external api that provided by dolphinscheduler, but we do not know how to use this feature(#79) exactly.
I think the commit record will help us but I can not find it in branch 1.0.2.

Which version of DolphinScheduler:
-[1.1.0-preview]

Additional context
Add any other context about the problem here.

**Requirement or improvement
Could you please list the link?
@zhaoyw1 zhaoyw1 added the question Further information is requested label Feb 12, 2020
@davidzollo
Copy link
Contributor

sorry for reply so late, this issue already solved or not ?

@zhaoyw1
Copy link
Author

zhaoyw1 commented Mar 19, 2020 via email

@davidzollo
Copy link
Contributor

refer API DOC:
http://106.75.43.194:8888/dolphinscheduler/doc.html?language=zh_CN&lang=zh

@davidzollo
Copy link
Contributor

refer API DOC:
http://106.75.43.194:8888/dolphinscheduler/doc.html?language=zh_CN&lang=zh

this is 1.2.0 API doc, maybe there is little difference with 1.1.0

@zhaoyw1
Copy link
Author

zhaoyw1 commented Apr 17, 2020 via email

@davidzollo
Copy link
Contributor

Hi, how are you? Last email you provided the API DOC for dolphin scheduler, and it helps us a lot. Thx sincerely! But when I use ProcessInstanceController#updateProcessInstance, the param "scheduler time" cause some trouble. Here is the command how I tested: curl -X POST "http://incubator-t3-infra04:12345/dolphinscheduler/projects/test_api/instance/update?processInstanceJson=&processInstanceId=3&scheduleTime=&syncDefine=&locations=&connects=&flag=NO" -H "token:62ec76d988f746303226fe4216fd6247" (the token belong to admin and doesnt expried) Here is the response:{"code":0,"msg":"success","data":null} While the fact is the corresponding process instances flag doesn`t update to 0(from 1,get verfied through t_ds_process_definition), and check the log of dolphinscheduler-api-server.log, find that: [INFO] 2020-04-17 10:15:08.797 org.apache.dolphinscheduler.api.controller.ProcessInstanceController:[178] - updateProcessInstance process instance, login user:admin, project name:test_api, process instance json:,process instance id:3, schedule time:, sync define:false, flag:NO, locations:, connects: [ERROR] 2020-04-17 10:15:08.800 org.apache.dolphinscheduler.common.utils.DateUtils:[123] - error while parse date: java.time.format.DateTimeParseException: Text '' could not be parsed at index 0 at java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1949) at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1851) at java.time.LocalDateTime.parse(LocalDateTime.java:492) at org.apache.dolphinscheduler.common.utils.DateUtils.parse(DateUtils.java:120) at org.apache.dolphinscheduler.common.utils.DateUtils.stringToDate(DateUtils.java:136) at org.apache.dolphinscheduler.common.utils.DateUtils.getScheduleDate(DateUtils.java:220) at org.apache.dolphinscheduler.api.service.ProcessInstanceService.updateProcessInstance(ProcessInstanceService.java:375) at org.apache.dolphinscheduler.api.service.ProcessInstanceService$$FastClassBySpringCGLIB$$da33359.invoke() in ProcessInstanceService.updateProcessInstance, the releated code is: So I am wondering whether "!schedulerTime.isEmpty()" could solve this error for ".scheduleTime=&." situation. Yours sincerely.
…
------------------------------------------------------------------ 发件人：dailidong notifications@github.com 发送时间：2020年3月21日(星期六) 16:42 收件人：apache/incubator-dolphinscheduler incubator-dolphinscheduler@noreply.github.com 抄　送：赵玉威 zhaoyw@t3go.cn; Author author@noreply.github.com 主　题：Re: [apache/incubator-dolphinscheduler] [QUESTION]About the commit record for issues#79 (#1945) refer API DOC: http://106.75.43.194:8888/dolphinscheduler/doc.html?language=zh_CN&lang=zh this is 1.2.0 API doc, maybe there is little difference with 1.1.0 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

did you solve this problem?

@davidzollo
Copy link
Contributor

it's a long time, please re-submit an issue if you meet any problem

@zhaoyw1
Copy link
Author

zhaoyw1 commented Nov 16, 2020 via email

@davidzollo
Copy link
Contributor

Yes, have solved it by what I mentioned in this E-mail. Besides, we have launched lots of hudi tasks(spark) which are scheduled frequently for incremental synchronization. But every time they are scheduled, they will download the same jar-resources and then "not enough disk space" alert are triggered frequently. So I realize the resource reference and synchronization, this repetitive and unnecessary step has been avoided. Later I will share the solution in E-mail and hope it useful for community if no related pr is merged. 这个问题已经解决了，用的就是我在邮件里面提到的方式。 此外，我们近期上了大批量的调度非常频繁的hudi增量同步任务，由于每次任务都会从hdfs上拉取资源到本地，磁盘空间不足的告警会被频繁触发。所以我实现了资源引用与同步来解决这个问题，目前生产上已经不存在这个问题了。最近我会把我的实现方式通过邮件的方式发出来，如果社区还没有类似的pr提交的话，希望有所帮助。
…
------------------------------------------------------------------ 发件人：dailidong notifications@github.com 发送时间：2020年11月16日(星期一) 16:21 收件人：apache/incubator-dolphinscheduler incubator-dolphinscheduler@noreply.github.com 抄　送：赵玉威 zhaoyw@t3go.cn; Author author@noreply.github.com 主　题：Re: [apache/incubator-dolphinscheduler] [QUESTION]About the commit record for issues#79 (#1945) Hi, how are you? Last email you provided the API DOC for dolphin scheduler, and it helps us a lot. Thx sincerely! But when I use ProcessInstanceController#updateProcessInstance, the param "scheduler time" cause some trouble. Here is the command how I tested: curl -X POST "http://incubator-t3-infra04:12345/dolphinscheduler/projects/test_api/instance/update?processInstanceJson=&processInstanceId=3&scheduleTime=&syncDefine=&locations=&connects=&flag=NO" -H "token:62ec76d988f746303226fe4216fd6247" (the token belong to admin and doesnt expried) Here is the response:{"code":0,"msg":"success","data":null} While the fact is the corresponding process instances flag doesn`t update to 0(from 1,get verfied through t_ds_process_definition), and check the log of dolphinscheduler-api-server.log, find that: [INFO] 2020-04-17 10:15:08.797 org.apache.dolphinscheduler.api.controller.ProcessInstanceController:[178] - updateProcessInstance process instance, login user:admin, project name:test_api, process instance json:,process instance id:3, schedule time:, sync define:false, flag:NO, locations:, connects: [ERROR] 2020-04-17 10:15:08.800 org.apache.dolphinscheduler.common.utils.DateUtils:[123] - error while parse date: java.time.format.DateTimeParseException: Text '' could not be parsed at index 0 at java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1949) at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1851) at java.time.LocalDateTime.parse(LocalDateTime.java:492) at org.apache.dolphinscheduler.common.utils.DateUtils.parse(DateUtils.java:120) at org.apache.dolphinscheduler.common.utils.DateUtils.stringToDate(DateUtils.java:136) at org.apache.dolphinscheduler.common.utils.DateUtils.getScheduleDate(DateUtils.java:220) at org.apache.dolphinscheduler.api.service.ProcessInstanceService.updateProcessInstance(ProcessInstanceService.java:375) at org.apache.dolphinscheduler.api.service.ProcessInstanceService$$FastClassBySpringCGLIB$$da33359.invoke() in ProcessInstanceService.updateProcessInstance, the releated code is: So I am wondering whether "!schedulerTime.isEmpty()" could solve this error for ".scheduleTime=&." situation. Yours sincerely. … ------------------------------------------------------------------ 发件人：dailidong notifications@github.com 发送时间：2020年3月21日(星期六) 16:42 收件人：apache/incubator-dolphinscheduler incubator-dolphinscheduler@noreply.github.com 抄　送：赵玉威 zhaoyw@t3go.cn; Author author@noreply.github.com 主　题：Re: [apache/incubator-dolphinscheduler] [QUESTION]About the commit record for issues#79 (#1945) refer API DOC: http://106.75.43.194:8888/dolphinscheduler/doc.html?language=zh_CN&lang=zh this is 1.2.0 API doc, maybe there is little difference with 1.1.0 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. did you solve this problem? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

very good idea，looking forwarding for your contribution

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@davidzollo @zhaoyw1