Permalink
Browse files

(doc) update wiki on rate limiting / vsts feeds

  • Loading branch information...
ferventcoder committed Dec 4, 2018
1 parent 79025c2 commit 1c1301092953d77a4f78c951bfd9f2d5de594b4a
Submodule choco.wiki updated from d2e4e8 to e9f31a
@@ -49,15 +49,16 @@
<p><strong>Blocks are meant to be temporary bans, but require you to act to remedy the situation.</strong> If you have been blocked, please see the next sections for corrective actions.</p>
<p><strong>NOTE</strong>: If you or your organization feels you will need to go over this limit and need whitelisted, please reach out at <a href="https://chocolatey.org/contact" class="uri">https://chocolatey.org/contact</a>, choose &quot;Blocked IP Address&quot;. As we have limited information, please include your name, email address, phone number, and the IP addresses you believe are blocked so we can contact you and verify if there is a ban. Once you have resolved any issues on your side, we can lift the ban.</p>
<h3 id="how-to-avoid-excessive-use">How To Avoid Excessive Use</h3>
<p>To avoid excessive use, please see our <a href="@Url.RouteUrl(RouteName.Docs, new { docName = "how-to-setup-offline-installation" })">organizational deployment guide</a>. There are even ways to automate caching / <a href="@Url.RouteUrl(RouteName.Docs, new { docName = "how-to-recompile-packages" })">internalizing</a> (caching and internalizing are entirely different concepts) packages so you still get a pretty good hands off experience.</p>
<p>To avoid excessive use, please see our <a href="@Url.RouteUrl(RouteName.Docs, new { docName = "how-to-setup-offline-installation" })">organizational deployment guide</a>. Installation of Chocolatey itself and everything else should be from your internal repository and not directly from the community package repository. There are even ways to automate caching (see below) / <a href="@Url.RouteUrl(RouteName.Docs, new { docName = "how-to-recompile-packages" })">internalizing</a> (caching and internalizing are entirely different concepts) packages so you still get a pretty good hands off experience.</p>
<p>If you are not able to take advantage of <a href="@Url.RouteUrl(RouteName.Docs, new { docName = "how-to-recompile-packages" })">internalizing</a> packages, you can still cache them locally (using package repository solutions like Artifactory, Nexus, ProGet, MyGet, etc), which will reduce your direct usage of the community repository. <strong>NOTE:</strong> Caching doesn&#39;t make the packages you are using from the community repository any more reliable, they may still need to download things from the internet at runtime - but it doesn&#39;t put you in a worse place than you already are at because you are already using the community repository directly which has issues identified in this document. If you want to achieve reliability when reusing community packages, you would need to <a href="@Url.RouteUrl(RouteName.Docs, new { docName = "how-to-recompile-packages" })">internalize packages</a>.</p>
<p>For caching of packages, something can be quickly implemented in 15–30 minutes to get your organization unblocked (and avoid rate limiting) while you look into implementing the rest of the <a href="@Url.RouteUrl(RouteName.Docs, new { docName = "how-to-setup-offline-installation" })">organizational deployment guide</a> (which takes about 1–2 hours). With 15–30 minutes, you can implement a <a href="https://help.sonatype.com/repomanager3/.net-package-repositories-with-nuget">Proxy Repository</a> including the install of a <a href="https://chocolatey.org/packages/nexus-repository">Nexus Repository Manager v3</a> (or <a href="https://chocolatey.org/packages/nexus-oss">NXRM v2</a>) which automatically caches (but does not <a href="@Url.RouteUrl(RouteName.Docs, new { docName = "how-to-recompile-packages" })">internalize</a>) packages from the community repository (<code>https://chocolatey.org/api/v2</code>). This provides the same experience you get in using the community repository now but with more availability and no rate limiting!</p>
<h3 id="what-to-do-when-you-are-blocked-for-excessive-use">What To Do When You Are Blocked For Excessive Use</h3>
<p><strong>NOTE: A block will not automatically expire, you will need to contact our team to resolve the block.</strong></p>
<p><strong>NOTE: A block will not automatically expire, you will need to contact our team to resolve the block.</strong> Rate Limiting on the other hand does automatically expire after one hour. Please see <a href="#rate-limiting">rate limiting</a> below.</p>
<p>If you have found that you have gone over the limit and have been warned/blocked, please reach out at <a href="https://chocolatey.org/contact" class="uri">https://chocolatey.org/contact</a> (send message to &quot;Blocked IP Address&quot; in the drop down - you may need to do this from a different IP address) or go to <a href="https://gitter.im/chocolatey/choco" class="uri">https://gitter.im/chocolatey/choco</a> to contact the community team. You can also send an email to community at chocolatey dot io. As we have limited information (only an IP address), please include your name, email address, phone number, and the IP addresses you believe are blocked so we can contact you and verify if there is a block.</p>
<p>See the section above on avoiding excessive use - the expectation is that organizations would not use the community repository directly. As part of addressing any misconfigurations you might have, you will also need to see about addressing the previous section on &quot;How To Avoid Excessive Use&quot;.</p>
<p>Once you have resolved any issues on your side, we can lift the block. A block will be reimplemented later if we find excessive use again.</p>
<h3 id="rate-limiting">Rate Limiting</h3>
<p>As a measure to increase site stability and prevent excessive use, the Chocolatey website uses rate limiting on requests for the community repository. Most folks typically won&#39;t hit rate limits unless they are automatically tagged for excessive use. If you do trigger the rate limit, you will see a <code>(429) Too Many Requests</code>. When attempting to install Chocolatey you will see the following:</p>
<p>As a measure to increase site stability and prevent excessive use, the Chocolatey website uses rate limiting on requests for the community repository. Rate limiting was introduced in November 2018. Most folks typically won&#39;t hit rate limits unless they are automatically tagged for excessive use. If you do trigger the rate limit, you will see a <code>(429) Too Many Requests</code>. When attempting to install Chocolatey you will see the following:</p>
<p><img src="@Url.Content("~/content/images/docs/cloudflare_ratelimiting_choco_install.png")" alt="Exception calling DownloadFile with 2 arguments: The remote server returned an error: 429 Too Many Requests" /></p>
<p>It will look like the following using choco.exe:</p>
<p><img src="@Url.Content("~/content/images/docs/cloudflare_ratelimiting_choco.png")" alt="The remote server returned an error: 429 Too Many Requests" /></p>
@@ -71,13 +72,17 @@
<p><strong><em>NOTE: Please note that individuals using the community repository are unlikely to hit rate limiting under normal usage scenarios.</em></strong></p>
<p><strong>Details:</strong></p>
<ul>
<li>Package downloads/installations are rate limited at about 20 per minute per IP address.</li>
<li>Chocolatey package (chocolatey.nupkg) downloads/installations are rate limited at 5 per minute per IP address.</li>
<li>Installations/downloads of Chocolatey itself (chocolatey.nupkg) are rate limited at about 5 per minute per IP address - temporary ban expires after 1 hour.</li>
<li>All other packages are rate limited at about 20 per minute per IP address - temporary ban expires after 1 hour.</li>
</ul>
<p><strong>NOTE:</strong> Rate Limiting defaults are subject to change with or without notice as we find a good happy medium that ensures ease of use and stability for our community.</p>
<h3 id="what-to-do-when-you-are-rate-limited">What To Do When You Are Rate Limited</h3>
<p><strong>NOTE: A rate limit will automatically expire after an hour, but if you hit the limit again, it will block for another hour.</strong></p>
<p>If you have found that you have been rate limited, please see <a href="#how-to-avoid-excessive-use">How To Avoid Excessive Use</a>. Implementing best practices for organizational use will limit chances of being rate limited again in the future.</p>
<ul>
<li>Individual users being rate limited should reach out as per the next section and let us know as we are constantly adjusting limits to find a happy medium and need to have as much data to work with as possible. In addition to providing the requested information, make sure to also mention you are &quot;individual use&quot; and provide details on what caused the rate limiting. We may ask you to provide logs for further inspection.</li>
<li>Organizational use will be asked to set up best practices for Chocolatey deployments.</li>
</ul>
<h4 id="special-requests-on-rate-limiting">Special Requests on Rate Limiting</h4>
<p>If you have special needs and are being rate limited, please reach out to us as in special instances, we can whitelist your IP address for a small period of time. Do the following:</p>
<ul>
@@ -86,6 +86,7 @@
<li>NuGet.Server</li>
<li><a href="https://github.com/NuGet/NuGetGallery/wiki/Hosting-the-NuGet-Gallery-Locally-in-IIS">NuGet Gallery</a></li>
<li>Chocolatey Gallery</li>
<li>Visual Studio Team Services (use NuGet v2 urls)</li>
</ul>
<h4 id="others">Others</h4>
<p>These are other known servers, but we don&#39;t know the quality or compatibility of these frameworks as they relate to Chocolatey packages.</p>
@@ -161,6 +162,7 @@
<li><a href="https://github.com/MPIB/hazel">Hazel</a></li>
<li><a href="https://github.com/ikkentim/LaGet">LaGet</a></li>
<li><a href="https://github.com/themotleyfool/Klondike">Klondike</a></li>
<li>Visual Studio Team Services (use NuGet v2 urls)</li>
</ul>
<p>If the option you&#39;ve chosen is not listed, take a look at <a href="#known-hosting-options">known hosting options</a> and follow the link from there to see what the vendor provides in the way of documentation.</p>
<h3 id="advantages-1">Advantages</h3>

0 comments on commit 1c13010

Please sign in to comment.