CDNSource: Use a more compact, sharded index retrieving for the pod versions #541
Conversation
igor-makarov
force-pushed
the
cdn-repo-huge-index
branch
from
bc06607 to
46a5bff
Compare
igor-makarov
changed the title
igor-makarov
changed the title
igor-makarov
force-pushed
the
cdn-repo-huge-index
branch
from
46a5bff to
fffa3e8
Compare
|
I've been talking to Netlify about an open source license and they seem to be okay with hosting the CocoaPods spec repo cache. This is their open source program page. Additionally, proxying Netlify through jsDelivr would work as well. I've talked to them as well and they handle all the specifics correctly. |
| @@ -0,0 +1,2 @@ | |||
| BeaconKit/1.0.0/1.0.1/1.0.2/1.0.3/1.0.4/1.0.5 | |||
| SDWebImage/2.4/2.5/2.6/2.7/2.7.4/3.0/3.1/4.0.0/4.0.0-beta/4.0.0-beta2 | |||
There was a problem hiding this comment.
I like this version better than the previous draft since its a bit more compact 👍
There was a problem hiding this comment.
can Pods have '/' in their name? I'm not 100% if they can, but we might have to think about escaping things here
There was a problem hiding this comment.
As far as I managed to find, there are no pods with slashes. While theoretically possible, it's pretty hard to create a file with a slash in the name.
I don't know if pod linting checks for it.
There was a problem hiding this comment.
Also, I think Pod/Name is reserved for subspecs.
|
@amorde Keep in mind that while being more compact and performant (entire zip is only 750KB), using it will generate more CDN traffic, as each podspec publish will cause the entire index to get dirty. |
|
Yup was actually referring to this PR when it was a draft and had the compact index but the versions all on separate lines. Good point though on the traffic. Maybe there's a middle ground in which we separate this into smaller chunks, but not as many chunks as before. What if we did something similar to the spec repo sharing, and had an index per starting character (ex. a-pods.txt, b-pods.txt, 1-pods.txt)? |
|
Considering the random-like nature of the shard hash function, and the fact that most projects have multiple pods, I don't think sharding the compact index would help much. Unless you shard it all the way, but that might cause problems when building on Netlify. Ultimately, this index will be downloaded only when creating and updating a repo. Additionally, jsDelivr said they don't have a problem being the CDN in front of Netlify. |
igor-makarov
force-pushed
the
cdn-repo-huge-index
branch
from
fffa3e8 to
8a9c5a0
Compare
|
Alright! I've tested deploying full sharded indexes on Netlify and it works okay. Turns out they don't have a problem with ~4k files, only ~700k files. 🙃 |
igor-makarov
force-pushed
the
cdn-repo-huge-index
branch
from
8a9c5a0 to
e7849c4
Compare
igor-makarov
changed the title
|
Now that Netlify deployment is up, this can be merged safely. |
igor-makarov
changed the title
|
I've updated the original PR comment to reflect the code that's being PR'd. |
|
Awesome! 💪 |
|
Cherry-picked to |
So this is the new iteration of CDNSource. It no longer relies on per-pod
index.txtfiles, but similar to RubyGems, utilizes concentrated version index files.The per-shard files are named
all_pods_versions_9_a_4.txtand are assumed to be located in the root of the repo. The content of each file is of the following format:This change reduces the amount of deployment work down to 2 minutes on Netlify and there are as many files as there are shards (4,096).
I've also added to
CDNSourcethe capability to follow redirects.I've updated the tests and I believe new tests aren't necessary at this moment.
Netlify deployment has been completed and is possible and running.