-
Notifications
You must be signed in to change notification settings - Fork 290
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature Request] httpreader分页功能 #841
Comments
这个的难点在于目前界面分页不存在一个统一的分页标准,请求方式不同(有的用 GET, 有的用 POST),请求的参数更是五花八门,估计要实现也是会较多的限制条件下完成 |
就普通GET请求而言,参数只有开始时间、结束时间、页号、页大小这四个,想把几千万行记录遍历读完呢? |
wgzhao
added a commit
that referenced
this issue
Jul 12, 2023
wgzhao
added a commit
that referenced
this issue
Jul 13, 2023
wgzhao
added a commit
that referenced
this issue
Nov 6, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
@wgzhao 作者您好,目前addax里httpreader暂不支持分页。比如数据提供方提供了一个totalSize几千万行的分页查询接口,要把这几千万行同步到mysql,在httpreader里支持分页是否可行?还是说应该从外部给httpreader多次传分页参数,多次触发该同步任务?
@weihebu 看了您的
wiki
https://github.com/zhzava/DOC/wiki/datax-httpheader%E6%8F%92%E4%BB%B6%E4%BB%8B%E7%BB%8D 似乎是直接在httpreader里实现分页获取数据功能,是否可以只触发一次同步任务,多次调用分页查询接口?The text was updated successfully, but these errors were encountered: