Social media URL validation doesn't work for Twitter URLs

[tgoyne]

🌈

• I've read and understood the CONTRIBUTING guidelines and have done my best effort to follow.

Twitter has recently started doing a thing where if you request a page using a browser user agent and no cookies, it replies with a 302 redirect to the same URL and some set-cookie headers. Making the same request again with the cookies then gives a 200 and the actual page contents, but HTTP.get_actual_url() ignores the cookies and only looks at the location header, resulting in a redirect loop. The end result is that pod validation gives WARN | url: The URL (https://twitter.com/realm) is not reachable. even though the URL works in a browser.

Some examples HTTP requests that give 200s:

$ curl 'https://twitter.com/realm'
$ curl 'https://twitter.com/realm' -H 'user-agent: Fake'
$  curl 'https://twitter.com/realm' -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36' -H 'cookie: rweb_optin=side_no_out'

HTTP requests that give 302s:

$ curl 'https://twitter.com/realm' -H 'user-agent: Safari'

[dnkoutso]

Sure! Could use a bug fix here. Want to make a PR?

[dnkoutso]

Start here https://github.com/CocoaPods/CocoaPods/blob/master/lib/cocoapods/validator.rb#L428-L438

[dnkoutso]

Actually the fix is probably here https://github.com/CocoaPods/Core/blob/master/lib/cocoapods-core/http.rb#L62-L77

[abbeycode]

@dnkoutso I took a stab at it with a couple PRs, modifying the two different files you mentioned above. #9053 depends on the CocoaPods/Core#571 PR

I'm motivated by one of my builds breaking because of this issue, which is why I targeted my PRs to the 1.7 stable branches for each project. I'm guessing the build for #9053 won't succeed without the other being merged. Is there a way to mark a different branch or revision as a dependency?

[dnkoutso]

@abbeycode thanks, I dont think we will do a 1.7.6 for this, we are very close to 1.8.0.beta.1 so please target master on these PRs.

[paulb777]

In case it's helpful for others, here's the workaround we're using for Firebase CI - https://github.com/firebase/firebase-ios-sdk/blob/master/scripts/pod_lib_lint.rb#L75

[abbeycode]

@dnkoutso Is there a way to install CocoaPods into CI (Travis, for me) from master before the 1.8.0 beta is out?

[dnkoutso]

Yes using Bundler https://guides.cocoapods.org/using/a-gemfile.html and https://bundler.io/