[Sitemaps] Trim Unicode whitespace around URLs

[moviewang]

loc.toString().trim() didn' trim all whitespace

[sebastian-nagel]

Hi @moviewang, can you share a minimal sitemap to reproduce your problem? It's ok if the URLs are masked but the white space should be there. Thanks!
After a first look: String.trim() only removes ASCII white space and control characters and does not remove all Unicode white space. I see no issues replacing the trim() by a method which also strips Unicode white space, similar StringUtils.strip() from commons-lang.

[moviewang]

Hi @moviewang, can you share a minimal sitemap to reproduce your problem? It's ok if the URLs are masked but the white space should be there. Thanks!
After a first look: String.trim() only removes ASCII white space and control characters and does not remove all Unicode white space. I see no issues replacing the trim() by a method which also strips Unicode white space, similar StringUtils.strip() from commons-lang.

Yes, the sitemap's loc elements contains white space, but I'have no rights to modify it. Is there any else approach to solve the problem. Thanks!

[kkrugler]

Hi @moviewang - can you provide the URL to the sitemap?

[moviewang]

Hi@kkrugler - The sitemapIndex content like this.

[sebastian-nagel]

Hi @moviewang, is there any invisible Unicode white space (not in the ASCII range)? I've tried to reproduce it with a similar file:

• the first URL is properly trimmed
  
• the second isn't because there is a non-breaking space (U+00a0) hidden
  

[kkrugler]

Hi @moviewang - I tried modifying one of the existing sitemap index parsing tests to show this problem, but it seems to parse the URL without any issues (assuming they look like what you showed above). I did see some other issues, like the individual sitemaps not having their "processed" flag set, and no last modified date, etc. but that's a different can of worms.

[kkrugler]

@moviewang - the dates in your example aren't valid for sitemaps, then need to follow one of these formats. So if the dates are in UTC, they should be something like 2018-12-12T02:36:56Z.

There are two other issues, which I'll file as bugs separately, but only one might impact you. Data strings aren't getting trimmed, so the whitespace around the date string could cause problems.

[moviewang]

Hi @sebastian-nagel @kkrugler
Thanks a lot ,I'm very grateful to you for your help. the problem maybe caused by the the url contains unicode white space as @sebastian-nagel said. Please close this issue. Thanks!!!

[sebastian-nagel]

Hi @moviewang, I would suggest the parser also trim Unicode white space. I'll open a PR to fix this. The intention of my inquiry was to make sure whether this is the reason for your problem. Thanks!