Hitting captcha error for Zillow

[dimo414]

I've started hitting an error with the Zillow API:

ERROR   error while estimating valuation: estimator failed: error looking up property on zillow: error querying zillow api: {"appId":"PXHYx10rg3","jsClientSrc":"/HYx10rg3/init.js","firstPartyEnabled":true,"uuid":"fe855645-5fde-11ee-837c-59b695626650","hostUrl":"/HYx10rg3/xhr","blockScript":"/HYx10rg3/captcha/captcha.js?a=c&u=fe855645-5fde-11ee-837c-59b695626650&v=&m=0","altBlockScript":"https://captcha.px-cloud.net/PXHYx10rg3/captcha.js?a=c&u=fe855645-5fde-11ee-837c-59b695626650&v=&m=0","customLogo":"https://www.zillowstatic.com/s3/pfs/static/z-logo-default.svg"}:

I'm able to replicate the request to https://www.zillowstatic.com/autocomplete/v3/suggestions?q=... via curl successfully, so it looks like it's the second request to zm.zillow.com that is being rejected.

Not sure if there's any easy fixes here, but figured it was worth reporting at least.

[peppys]

Interesting. Is this happening every time for you now? I'm actually not able to recreate this. How often are you running this program? I wonder if they have some anti-spam automation kicking in.

[dimo414]

Not often at all, still just experimenting with crib. It's happening consistently on my Google Cloud instance atm, but it was working up until a day or so ago.

[peppys]

Hmm that's odd. I just spun up a GCP instance and tested it out, and it seems to work fine. Is it still happening to you, and does it happen consistently for any address you provide?

[dimo414]

Yeah hitting it regardless of the address, e.g.:

$ bin/crib value -a '1443 devlin dr, los angeles, ca'
 ██████ ██████  ██ ██████
██      ██   ██ ██ ██   ██
██      ██████  ██ ██████
██      ██   ██ ██ ██   ██
 ██████ ██   ██ ██ ██████

  ERROR   error while estimating valuation: estimator failed: error looking up property on zillow: error querying zillow api: {"appId":"PXHYx10rg3","jsClientSrc":"/HYx10rg3/init.js","firstPartyEnabled":true,"uuid":"73afb4ec-6725-11ee-a57c-5e091c608016","hostUrl":"/HYx10rg3/xhr","blockScript":"/HYx10rg3/captcha/captcha.js?a=c&u=73afb4ec-6725-11ee-a57c-5e091c608016&v=&m=0","altBlockScript":"https://captcha.px-cloud.net/PXHYx10rg3/captcha.js?a=c&u=73afb4ec-6725-11ee-a57c-5e091c608016&v=&m=0","customLogo":"https://www.zillowstatic.com/s3/pfs/static/z-logo-default.svg"}:

Rebooting the instance to get a new IP address similarly doesn't help.

[peppys]

Interesting! It seems others similarly get captcha'd when trying to query Zillow's private APIs from Google's servers.

https://stackoverflow.com/questions/41772899/google-app-engine-in-production-unable-to-talk-to-zillow-api-demands-captcha
https://stackoverflow.com/questions/41352056/zillow-api-asking-for-captcha

My guess is Zillow blocked a set of IP ranges to identify bots from GCP. Not sure what we can do here :/ This may not work in GCP for that reason.