Feature request: ipsw download ota --beta --pattern PATTERN (using theiphonewiki.com)

[glen-mac]

Hi! Have a small feature request. Similar to

$ ipsw download beta <build>

It would be awesome to be able to download Beta OTAs using links on the iphonewiki.com. On top of that being able to specify a pattern similar to the command:

$ ipsw download -d iPhone14,2 --latest --pattern iBoot

Would be such an awesome addition.

[blacktop]

Is there a reason why you want to download OTAs from theiphonewiki and not just from apple?

[glen-mac]

iiuc apple will remove the links from their xml listings for OTA betas as they do for IPSW betas - this is to download older versions that are no longer signed

[blacktop]

Hey so I took another look at the iphonewiki and it is VERY complex and I'm not really sure how to proceed.

For starters crawling their tables is a nightmare because there is 0 standardization and I EVEN am using the API now and not just scraping and it is still VERY gross to work with.

Also OTAs don't really lend themselves to being "categorized" let me show you what I mean - https://www.theiphonewiki.com/wiki/OTA_Updates/iPhone/15.0

How would you query those OTA zips? ipsw download wiki --version 15.0 --prereqver 14.0.1 --ota --device iPhone8,1 --board N71AP ???

I guess typically peeps would want the n/a n/a version of OTAs?

So it's parsing HTML tables with random spans everywhere OR parsing wikitable markup with random spans everywhere.

🤮

[blacktop]

It looks like the ipsw.me dude has old OTAs, maybe I could add support for those?

[blacktop]

I guess you were interested in BETA OTAs right? Which ipsw.me doesn't have 🤔

[blacktop]

This kinda makes me want to build my own version of ipsw.me 😬

[glen-mac]

heh yeah it looks disgusting to scrape

re: the deltas the n/a prerequisite is really the only one of interest

i’m surprised that ipsw.me doesn’t catalogue these, must really only have importance to researchers.

[glen-mac]

This just came to my attention if it is of any help in parsing.

https://www.theiphonewiki.com/w/index.php?title=Beta_Firmware/iPhone/14.x&action=raw

Noting the “raw” formatting.

[blacktop]

Yeah that is the format I'm consuming when I was talking to the API, I believe that is MediaWiki MarkUp format. However, it turned out to be almost equally annoying to work with due to the rowspan="5" I understand why the wiki authors do it, but it makes auto-parsing very difficult. You can see my initial attempts here - https://github.com/blacktop/ipsw/blob/master/internal/download/iphonewiki.go#L200

I'm not saying it's impossible, just that it is equally difficult due to having to keep track of the spans and then "repeating" the fields above to the following, what I'm calling, WikiIPSWs.

[blacktop]

FYI @glen-mac I added --pattern to the ipsw download ota cmd

[glen-mac]

ah lovely, thanks!

[zcutlip]

lol wish I'd seen this sooner. I actually did exactly this: I scrape the iPhone wiki, and I publish ipsw.me v2 compatible JSON at https://ipsw.inode.link/ipsw-beta.json, so you can grab full, non-OTA IPSWs.

Unfortunately the person who updates the wiki hasn't updated it since September, so my JSON feed isn't up to date either. :-(

[blacktop]

revamped the ipsw download wiki --ota --beta cmd to also allow you to filter by --pv prerequisite version and --pb prerequisite build (--pattern should still work) in latest release pushing out now (also it now parses the MediaWiki Markdown format exposed by the API