Releases: apify/crawlee
Releases · apify/crawlee
v0.16.0
v0.15.5
v0.15.4
- DEPRECATED:
dataset.delete()
,keyValueStore.delete()
andrequestQueue.delete()
methods have been deprecated in favor of*.drop()
methods, because thedrop
name more clearly communicates the fact that those methods drop / delete the storage itself, not individual elements in the storage. - Added
Apify.utils.requestAsBrowser()
helper function that enables you to make HTTP(S) requests disguising as a browser (Firefox). This may help in overcoming certain anti-scraping and anti-bot protections. - Added
options.gotoTimeoutSecs
toPuppeteerCrawler
to enable easier setting of navigation timeouts. PuppeteerPool
options that were deprecated from thePuppeteerCrawler
constructor were finally removed. Please usemaxOpenPagesPerInstance
,retireInstanceAfterRequestCount
,instanceKillerIntervalSecs
,killInstanceAfterSecs
andproxyUrls
via thepuppeteerPoolOptions
object.- On the Apify Platform a warning will now be printed when using an outdated
apify
package version. Apify.utils.puppeteer.enqueueLinksByClickingElements()
will now print a warning when the nodes it
tries to click become modified (detached from DOM). This is useful to debug unexpected behavior.
v0.15.3
Apify.launchPuppeteer()
now acceptsproxyUrl
with thehttps
,socks4
andsocks5
schemes, as long as it doesn't contain username or password.
This is to fix Issue #420.- Added
desiredConcurrency
option toAutoscaledPool
constructor, removed
unnecessary bound check from the setter property
v0.15.2
v0.15.1
- BREAKING CHANGE Removed support for Web Driver (Selenium) since no further updates are planned.
If you wish to continue using Web Driver, please stay on Apify SDK version ^0.14.15 - BREAKING CHANGE:
Dataset.getData()
throws an error if user provides an unsupported option
when using local disk storage. - DEPRECATED:
options.userData
ofApify.utils.enqueueLinks()
is deprecated.
Useoptions.transformRequestFunction
instead. - Improve logging of memory overload errors.
- Improve error message in
Apify.call()
. - Fix multiple log lines appearing when a crawler was about to finish.
- Add
Apify.utils.puppeteer.enqueueLinksByClickingElements()
function which enables you
to add requests to the queue from pure JavaScript navigations, form submissions etc. - Add
Apify.utils.puppeteer.infiniteScroll()
function which helps you with scrolling to the bottom
of websites that auto-load new content. - The
RequestQueue.handledCount()
function has been resurrected from deprecation,
in order to have compatible interface withRequestList
. - Add
useExtendedUniqueKey
option toRequest
constructor to includemethod
andpayload
in theRequest
's computeduniqueKey
. - Updated Puppeteer to 1.18.1
- Updated
apify-client
to 0.5.22