Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] httpreader分页功能 #841

Closed
dongjunchuan opened this issue Jul 12, 2023 · 2 comments · Fixed by #842
Closed

[Feature Request] httpreader分页功能 #841

dongjunchuan opened this issue Jul 12, 2023 · 2 comments · Fixed by #842

Comments

@dongjunchuan
Copy link

@wgzhao 作者您好,目前addax里httpreader暂不支持分页。比如数据提供方提供了一个totalSize几千万行的分页查询接口,要把这几千万行同步到mysql,在httpreader里支持分页是否可行?还是说应该从外部给httpreader多次传分页参数,多次触发该同步任务?

@weihebu 看了您的wikihttps://github.com/zhzava/DOC/wiki/datax-httpheader%E6%8F%92%E4%BB%B6%E4%BB%8B%E7%BB%8D 似乎是直接在httpreader里实现分页获取数据功能,是否可以只触发一次同步任务,多次调用分页查询接口?

@wgzhao
Copy link
Owner

wgzhao commented Jul 12, 2023

这个的难点在于目前界面分页不存在一个统一的分页标准,请求方式不同(有的用 GET, 有的用 POST),请求的参数更是五花八门,估计要实现也是会较多的限制条件下完成

@dongjunchuan
Copy link
Author

就普通GET请求而言,参数只有开始时间、结束时间、页号、页大小这四个,想把几千万行记录遍历读完呢?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants