Skip to content

feat: requestList accepts proxyConfiguration for requestsFromUrls#1317

Merged
B4nan merged 1 commit intoapify:masterfrom
barjin:feat/requestListProxy
Mar 28, 2022
Merged

feat: requestList accepts proxyConfiguration for requestsFromUrls#1317
B4nan merged 1 commit intoapify:masterfrom
barjin:feat/requestListProxy

Conversation

@barjin
Copy link
Copy Markdown
Member

@barjin barjin commented Mar 23, 2022

closes #1073

Intended use:

const proxyConfiguration = await Apify.createProxyConfiguration({ 
    // the proxy is to be specified here 
});

 const requestList = await Apify.openRequestList('test',[
        { requestsFromUrl:'http://sitemap.txt/sitemap.txt' }
    ], { proxyConfiguration }
 );

Copy link
Copy Markdown
Member

@B4nan B4nan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking good, please add at least one test that verifies the proxy is being actually used (can be just a unit test)

Comment thread src/request_list.js
* If `null`, the list will not be persisted and will only be stored in memory. Process restart
* will then cause the list to be crawled again from the beginning. We suggest always using a name.
* @param {Array<RequestOptions|Request|string>} sources
* @param {RequestListOptions['sources']} sources
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets see how this will work with the old docs (it should work fine with the sdk 3 ts api docs, but here - god knows)

@B4nan B4nan requested a review from pocesar March 28, 2022 10:01
Copy link
Copy Markdown
Contributor

@pocesar pocesar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @barjin !

Comment thread test/request_list.test.js
'http://another.proxy.url',
];

const mock = sinon.mock(utils.publicUtils);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dont forget to migrate this to jest when backporting (no need to do it now, we can backport larger chunk later)

@B4nan B4nan merged commit c120837 into apify:master Mar 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

requestsFromUrl should use proxies on RequestList/openRequestList

3 participants