Timezone of date when web scraping

[Chris3773]

I'm web scraping a website and it has as UTC datetime using this format 2023-06-20T14:05:00. My server is in EST so it renders the datetime as EST without converting from UTC to EST.

Is there a way to do this conversion within the XPath? Is the Custom date/time format what I'm looking for and if so what would I need to set to tell it is in UTC?

[Alkarex]

If you are using XPath, you could try something like concat(..., 'Z') to your expression that is getting the timestamp.

[Chris3773]

That did the trick. Thanks.

[matotermi1986]

Unfortunately, XPath alone won't handle the conversion for you. You'll need to handle it programmatically on your server-side code using a language like Python or JavaScript. Look into libraries like datetime in Python or moment.js in JavaScript to convert the datetime strings.

I recently had to deal with a similar issue while scraping data from a website. It was a lifesaver to find a service like https://www.nannostomus.com/data-harvesting/that offered data harvesting solutions. They made the whole process much smoother and saved me a lot of time and effort.

[Chris3773]

The server seems to handle converting from UTC to your server time zone. The issue I had was the time zone I was scaping was UTC but wasn't marked with Z stating it was UTC so the server thought it was in the server's time zone. So adding the Z to end of the timestamp fixed that issue.

[Charlotte-br560]

You can use Crawlbase to handle this conversion easily. Use the Custom date/time format feature and specify the input format as UTC. This will automatically convert the datetime to your server's timezone (EST) during web scraping.