cocoapods.py generates some JSON-related URLs that lead to a 404

[johnmhoran]

In connection with a purl2url issue in packageurl-python, I've been exploring the URL-related code in packagedcode's cocoapods.py. With the four PURL spec examples for cocoapods,

pkg:cocoapods/AFNetworking@4.0.1
pkg:cocoapods/MapsIndoors@3.24.0
pkg:cocoapods/ShareKit@2.0#Twitter
pkg:cocoapods/GoogleUtilities@7.5.2#NSData+zlib

I got the following results looking for potentially useful JSON files. Using the pattern

f'https://raw.githubusercontent.com/CocoaPods/Specs/blob/master/Specs/{hashed_path}/{name}/{version}/{name}.podspec.json'

from the get_urls() api_data_url variable, we get the following URLs, each of which leads to a 404: Not Found page:

• https://raw.githubusercontent.com/CocoaPods/Specs/blob/master/Specs/a/7/5/AFNetworking/4.0.1/AFNetworking.podspec.json
• https://raw.githubusercontent.com/CocoaPods/Specs/blob/master/Specs/f/c/a/MapsIndoors/3.24.0/MapsIndoors.podspec.json
• https://raw.githubusercontent.com/CocoaPods/Specs/blob/master/Specs/3/b/7/ShareKit/2.0/ShareKit.podspec.json
• https://raw.githubusercontent.com/CocoaPods/Specs/blob/master/Specs/0/8/4/GoogleUtilities/7.5.2/GoogleUtilities.podspec.json

A few lines above that pattern in get)url() is a pattern for the specs_json_cdn_url variable

f'https://cdn.cocoapods.org/Specs/{hashed_path}/{name}/{version}/{name}.podspec.json' 

For the same four cocoapods PURLs, this pattern generates valid URLs to cdn.cocoapods.org JSON files:

• https://cdn.cocoapods.org/Specs/a/7/5/AFNetworking/4.0.1/AFNetworking.podspec.json
• https://cdn.cocoapods.org/Specs/f/c/a/MapsIndoors/3.24.0/MapsIndoors.podspec.json
• https://cdn.cocoapods.org/Specs/3/b/7/ShareKit/2.0/ShareKit.podspec.json
• https://cdn.cocoapods.org/Specs/0/8/4/GoogleUtilities/7.5.2/GoogleUtilities.podspec.json

BTW, some of the URL data in these four .json files is not (or no longer) valid, e.g., the "homepage" URL for ShareKit (http://getsharekit.com/) leads to what might be a Turkish-language page -- https://smartem.org/. (The "source"/"git" URL (https://github.com/ShareKit/ShareKit.git) is valid and reflects that the last commit was made in December 2017.)

[pombredanne]

Remove /blob from the URLs:

• https://raw.githubusercontent.com/CocoaPods/Specs/master/Specs/a/7/5/AFNetworking/4.0.1/AFNetworking.podspec.json
• https://raw.githubusercontent.com/CocoaPods/Specs/master/Specs/f/c/a/MapsIndoors/3.24.0/MapsIndoors.podspec.json
• https://raw.githubusercontent.com/CocoaPods/Specs/master/Specs/3/b/7/ShareKit/2.0/ShareKit.podspec.json
• https://raw.githubusercontent.com/CocoaPods/Specs/master/Specs/0/8/4/GoogleUtilities/7.5.2/GoogleUtilities.podspec.json

[pombredanne]

So use instead:
f'https://raw.githubusercontent.com/CocoaPods/Specs/master/Specs/{hashed_path}/{name}/{version}/{name}.podspec.json'

[pombredanne]

BTW, some of the URL data in these four .json files is not (or no longer) valid,

This would be another issue entirely ... We should have separate ways to crawl and tag invalid or dead URLs, and this would be implemented likely in the PurlDB, as some improver.