Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CDNSource: Use a more compact, sharded index retrieving for the pod versions #541

Merged
merged 2 commits into from Jun 12, 2019

Conversation

Projects
None yet
4 participants
@igor-makarov
Copy link
Contributor

commented May 21, 2019

So this is the new iteration of CDNSource. It no longer relies on per-pod index.txt files, but similar to RubyGems, utilizes concentrated version index files.

The per-shard files are named all_pods_versions_9_a_4.txt and are assumed to be located in the root of the repo. The content of each file is of the following format:

PodName/1.0.3/1.1/1.1.1/1.1.2/1.1.3/1.1.4

This change reduces the amount of deployment work down to 2 minutes on Netlify and there are as many files as there are shards (4,096).

I've also added to CDNSource the capability to follow redirects.

I've updated the tests and I believe new tests aren't necessary at this moment.

Netlify deployment has been completed and is possible and running.

@igor-makarov igor-makarov force-pushed the igor-makarov:cdn-repo-huge-index branch from bc06607 to 46a5bff May 22, 2019

@igor-makarov igor-makarov changed the title Use a more compact, one-file index for all the pods Use a more compact, one-file index for all the pods - Netlify deployment is possible! May 22, 2019

@igor-makarov igor-makarov marked this pull request as ready for review May 22, 2019

@igor-makarov igor-makarov reopened this May 22, 2019

@igor-makarov igor-makarov changed the title Use a more compact, one-file index for all the pods - Netlify deployment is possible! CDNSource: Use a more compact, one-file index for all the pods - Netlify deployment is possible! May 22, 2019

@igor-makarov igor-makarov force-pushed the igor-makarov:cdn-repo-huge-index branch from 46a5bff to fffa3e8 May 27, 2019

@igor-makarov

This comment has been minimized.

Copy link
Contributor Author

commented May 28, 2019

I've been talking to Netlify about an open source license and they seem to be okay with hosting the CocoaPods spec repo cache.

This is their open source program page.

Additionally, proxying Netlify through jsDelivr would work as well. I've talked to them as well and they handle all the specifics correctly.

@@ -0,0 +1,2 @@
BeaconKit/1.0.0/1.0.1/1.0.2/1.0.3/1.0.4/1.0.5
SDWebImage/2.4/2.5/2.6/2.7/2.7.4/3.0/3.1/4.0.0/4.0.0-beta/4.0.0-beta2

This comment has been minimized.

Copy link
@amorde

amorde May 29, 2019

Member

I like this version better than the previous draft since its a bit more compact 👍

This comment has been minimized.

Copy link
@amorde

amorde May 29, 2019

Member

can Pods have '/' in their name? I'm not 100% if they can, but we might have to think about escaping things here

This comment has been minimized.

Copy link
@igor-makarov

igor-makarov May 29, 2019

Author Contributor

As far as I managed to find, there are no pods with slashes. While theoretically possible, it's pretty hard to create a file with a slash in the name.

I don't know if pod linting checks for it.

This comment has been minimized.

Copy link
@igor-makarov

igor-makarov May 29, 2019

Author Contributor

Also, I think Pod/Name is reserved for subspecs.

This comment has been minimized.

Copy link
@amorde

amorde May 29, 2019

Member

Yeah that sounds right

@igor-makarov

This comment has been minimized.

Copy link
Contributor Author

commented May 29, 2019

@amorde Keep in mind that while being more compact and performant (entire zip is only 750KB), using it will generate more CDN traffic, as each podspec publish will cause the entire index to get dirty.

@amorde

This comment has been minimized.

Copy link
Member

commented May 29, 2019

Yup was actually referring to this PR when it was a draft and had the compact index but the versions all on separate lines. Good point though on the traffic. Maybe there's a middle ground in which we separate this into smaller chunks, but not as many chunks as before.

What if we did something similar to the spec repo sharing, and had an index per starting character (ex. a-pods.txt, b-pods.txt, 1-pods.txt)?

@igor-makarov

This comment has been minimized.

Copy link
Contributor Author

commented May 29, 2019

Considering the random-like nature of the shard hash function, and the fact that most projects have multiple pods, I don't think sharding the compact index would help much. Unless you shard it all the way, but that might cause problems when building on Netlify.

Ultimately, this index will be downloaded only when creating and updating a repo. Additionally, jsDelivr said they don't have a problem being the CDN in front of Netlify.

@igor-makarov igor-makarov force-pushed the igor-makarov:cdn-repo-huge-index branch from fffa3e8 to 8a9c5a0 May 30, 2019

@igor-makarov

This comment has been minimized.

Copy link
Contributor Author

commented May 30, 2019

Alright! I've tested deploying full sharded indexes on Netlify and it works okay. Turns out they don't have a problem with ~4k files, only ~700k files. 🙃
This new version has a text file for each shard, named like all_pods_versions_9_a_4.txt where the suffix is the shard path. Each file is around 1KB, and while my total (for a project with 30 pods) was 300KB - these aren't expected to churn as much!

@igor-makarov igor-makarov force-pushed the igor-makarov:cdn-repo-huge-index branch from 8a9c5a0 to e7849c4 Jun 8, 2019

@igor-makarov igor-makarov changed the title CDNSource: Use a more compact, one-file index for all the pods - Netlify deployment is possible! CDNSource: Use a more compact, sharded index for the pods' versions Jun 10, 2019

@igor-makarov

This comment has been minimized.

Copy link
Contributor Author

commented Jun 10, 2019

Now that Netlify deployment is up, this can be merged safely.

@igor-makarov igor-makarov changed the title CDNSource: Use a more compact, sharded index for the pods' versions CDNSource: Use a more compact, sharded index retrieving for the pod versions Jun 10, 2019

@igor-makarov

This comment has been minimized.

Copy link
Contributor Author

commented Jun 10, 2019

I've updated the original PR comment to reflect the code that's being PR'd.

@dnkoutso dnkoutso added this to the 1.8 milestone Jun 11, 2019

@amorde

amorde approved these changes Jun 11, 2019

@orta

orta approved these changes Jun 12, 2019

@amorde amorde merged commit 0e8a708 into CocoaPods:master Jun 12, 2019

3 checks passed

Danger All good
Details
Peril All green. Yay.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@igor-makarov

This comment has been minimized.

Copy link
Contributor Author

commented Jun 12, 2019

Awesome! 💪

@igor-makarov igor-makarov deleted the igor-makarov:cdn-repo-huge-index branch Jun 13, 2019

@dnkoutso

This comment has been minimized.

Copy link
Contributor

commented Jun 13, 2019

Cherry-picked to 1-7-stable here.

@dnkoutso dnkoutso modified the milestones: 1.8, 1.7 Jun 13, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.