Add: switch to cdn.openttd.org for all downloads

Here also is a file called "latest.yaml", which greatly simplifies
the "fetch-downloads.py". Additionally, as cdn.openttd.org is
browseable, we only need to serve "latest" (and no longer older
releases).

While at it, refactored fetch_downloads a bit to be a Python
module and run flake8/black on it.

.github/workflows/testing.yml

@@ -19,3 +19,30 @@ jobs:
         with-submodules: true
     - name: Build
       run: docker build .
+
+  flake8:
+    name: Flake8
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+    - name: Flake8
+      uses: TrueBrain/actions-flake8@master
+      with:
+        path: fetch_downloads
+
+  black:
+    name: Black
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+    - name: Set up Python 3.8
+      uses: actions/setup-python@v1
+      with:
+        python-version: 3.8
+    - name: Black
+      run: |
+        python -m pip install --upgrade pip
+        pip install black
+        black -l 120 --check fetch_downloads

Dockerfile

@@ -12,12 +12,8 @@ RUN pip freeze 2>/dev/null | grep -v "deployer" > requirements.installed \
         && diff -u requirements.txt requirements.installed 1>&2 \
         || ( echo "!! ERROR !! requirements.txt defined different packages or versions for installation" \
                 && exit 1 ) 1>&2
-RUN pip install flake8
 
-COPY .flake8 .flake8
-RUN flake8 fetch-downloads.py
-
-RUN python fetch-downloads.py
+RUN python -m fetch_downloads
 
 # Build the HTML from the source
 FROM alpine as html

README.md

@@ -22,15 +22,13 @@ This is a [Jekyll](https://jekyllrb.com/) website, and is served by nginx as a s
 ### Populating _downloads
 
 By default `_downloads` is empty, so when building the website locally there is no latest stable/nightly.
-This can be resolved by running `fetch-downloads.py`.
+This can be resolved by running `fetch_downloads`.
 This script will download the latest available binaries, and populate `_downloads`.
 
-`fetch-downloads.py` is a Python3.6+ application, and will make ~400 HTTP connections to various of OpenTTD-related servers.
-
 ```bash
-python3 -m venv venv
-venv/bin/pip install -r requirements.txt
-venv/bin/python fetch-downloads.py
+python3 -m venv .env
+.env/bin/pip install -r requirements.txt
+.env/bin/python -m fetch_downloads
 ```
 
 ### Running a local server

_download-meta/catcodec-releases.md

@@ -1,3 +1,2 @@
 ---
-folder_old_infrastructure: extra/catcodec
 ---

_download-meta/grfcodec-nightlies.md

@@ -1,3 +1,2 @@
 ---
-folder_old_infrastructure: extra/grfcodec-trunk
 ---

_download-meta/grfcodec-releases.md

@@ -1,3 +1,2 @@
 ---
-folder_old_infrastructure: extra/grfcodec
 ---

_download-meta/nforenum-releases.md

@@ -1,5 +1,4 @@
 ---
-folder_old_infrastructure: extra/nforenum
 ---
 
 This package is deprecated. Please use [grfcodec](../grfcodec-releases/latest.html).

_download-meta/nosound-releases.md

@@ -1,3 +1,2 @@
 ---
-folder_old_infrastructure: extra/nosound
 ---

_download-meta/opengfx-releases.md

@@ -1,3 +1,2 @@
 ---
-folder_old_infrastructure: extra/opengfx
 ---

_download-meta/openmsx-releases.md

@@ -1,3 +1,2 @@
 ---
-folder_old_infrastructure: extra/openmsx
 ---

_download-meta/opensfx-releases.md

@@ -1,3 +1,2 @@
 ---
-folder_old_infrastructure: extra/opensfx
 ---

_download-meta/openttd-nightlies.md

@@ -1,5 +1,4 @@
 ---
-folder_old_infrastructure: nightlies/trunk
 ---
 
 OpenTTD requires a separate base graphics set to run, which can be downloaded through the installer (Windows) or on first run (Windows, Linux).
@@ -12,4 +11,4 @@ Sound and music sets are recommended (but not necessary).
 
 Additional graphics, sound, and music sets can be downloaded via the ingame content download manager.
 
-Please refer to the [readme](https://proxy.binaries.openttd.org/openttd-nightlies/@@version@@/README.md) for more information.
+Please refer to the [readme](https://cdn.openttd.org/@@folder@@/README.md) for more information.

_download-meta/openttd-pullrequests.md

@@ -1,5 +1,4 @@
 ---
-folder_old_infrastructure: releases
 ---
 
 OpenTTD requires a separate base graphics set to run, which can be downloaded through the installer (Windows) or on first run (Windows, Linux).
@@ -12,7 +11,7 @@ Sound and music sets are recommended (but not necessary).
 
 Additional graphics, sound, and music sets can be downloaded via the ingame content download manager.
 
-Please refer to the [readme](https://proxy.binaries.openttd.org/openttd-pullrequests/@@version@@/README.md) for more information.
+Please refer to the [readme](https://cdn.openttd.org/@@folder@@/README.md) for more information.
 
 **WARNING** - These binaries are built based on [Pull Request](https://github.com/OpenTTD/OpenTTD/pulls); the quality might vary, as the Pull Request might not be merge-ready.
 Savegames can be incompatible with other versions of OpenTTD and also with other versions of this Pull Request.

_download-meta/openttd-releases.md

@@ -1,5 +1,4 @@
 ---
-folder_old_infrastructure: releases
 ---
 
 OpenTTD requires a separate base graphics set to run, which can be downloaded through the installer (Windows) or on first run (Windows, Linux).
@@ -12,4 +11,4 @@ Sound and music sets are recommended (but not necessary).
 
 Additional graphics, sound, and music sets can be downloaded via the ingame content download manager.
 
-Please refer to the [readme](https://proxy.binaries.openttd.org/openttd-releases/@@version@@/README.md) for more information.
+Please refer to the [readme](https://cdn.openttd.org/@@folder@@/README.md) for more information.

_download-meta/openttd-useful.md → _download-meta/openttd-useful-releases.md

@@ -1,5 +1,4 @@
 ---
-folder_old_infrastructure: extra/openttd-useful
 ---
 
 **NOTE:**

_download-meta/osie-releases.md

@@ -1,3 +1,2 @@
 ---
-folder_old_infrastructure: extra/osie
 ---

_layouts/download.html

@@ -11,17 +11,7 @@
 {% assign raw_type = page.id | split: "/" | slice: 2 | first %}
 {% assign type = raw_type | prepend: "/download-meta/" %}
 {% assign meta = site.download-meta | where: "id", type | first %}
-{% unless page.version %}
-{% assign version = page.id | split: "/" | last %}
-{% else %}
-{% assign version = page.version %}
-{% endunless %}
-
-{% if page.on_old_infrastructure %}
-{% assign folder = meta.folder_old_infrastructure %}
-{% else %}
-{% assign folder = raw_type %}
-{% endif %}
+{% assign folder = page.version | prepend: "/" | prepend: page.folder %}
 
 {% unless meta.id %}
     {{ "No download-meta entry found for " | append: raw_type | raise_error }}
@@ -33,7 +23,7 @@ <h3>Download</h3>
     </div>
     <div class="section-item">
         <div class="content">
-            {{ meta.content | markdownify | replace: "@@version@@", version }}
+            {{ meta.content | markdownify | replace: "@@folder@@", folder }}
             <p>For all binaries officially released by us we publish the MD5, SHA1, and SHA256 checksums. You can use these checksums to check whether the file you downloaded has been modified. All three checksums should match the file you downloaded; if this is not the case it means that either the file didn't come from us or that it got broken during transport. Either way it might possibly contain dangerous modifications and the file should therefore not be trusted!</p>
         </div>
     </div>
@@ -42,9 +32,9 @@ <h3>Download {{ page.name }}</h3>
     </div>
     <div class="section-item">
         <div class="content">
-            <p>Latest release in {{ page.name }} is {{ version }}, released on {{ page.date | date: "%Y-%m-%d %H:%M" }} UTC.</p>
+            <p>Latest {{ page.name }} release in {{ page.category }} is {{ page.version }}, released on {{ page.date | date: "%Y-%m-%d %H:%M" }} UTC.</p>
 
-            <div class="changelog">[&nbsp;<a href="{{ page.host }}/{{ folder }}/{{ version }}/changelog.txt">Changelog</a>&nbsp;]</div>
+            <div class="changelog">[&nbsp;<a href="https://cdn.openttd.org/{{ folder }}/changelog.txt">Changelog</a>&nbsp;]</div>
             <div id="download-combo">
                 <input type="hidden" id="download-combo-state" value="" />
                 <input type="hidden" id="download-base-name" value="{{ page.base }}" />
@@ -59,7 +49,7 @@ <h3>Download {{ page.name }}</h3>
                 {% endunless %}
 
                 <li id="{{ file.id }}">
-                    <div class="filename"><a href="{{ page.host }}/{{ folder }}/{{ version }}/{{ file.id }}">{{ description }}</a></div>
+                    <div class="filename"><a href="https://cdn.openttd.org/{{ folder }}/{{ file.id }}">{{ description }}</a></div>
                     <div class="filesize">[&nbsp;{{ file.size | string_of_size }}&nbsp;]</div>
                     <div class="checksums-dropdown" onclick="toggleChecksum(this, 'checksum-{{ file.id }}')">[&nbsp;<a href="#" onclick="return false;">Checksums</a>&nbsp;]</div>
                     <div class="checksums" id="checksum-{{ file.id }}">
@@ -91,7 +81,7 @@ <h3>Developer Files</h3>
                 {% endunless %}
 
                 <li id="{{ file.id }}">
-                    <div class="filename"><a href="{{ page.host }}/{{ folder }}/{{ version }}/{{ file.id }}">{{ description }}</a></div>
+                    <div class="filename"><a href="https://cdn.openttd.org/{{ folder }}/{{ file.id }}">{{ description }}</a></div>
                     <div class="filesize">[&nbsp;{{ file.size | string_of_size }}&nbsp;]</div>
                     <div class="checksums-dropdown" onclick="toggleChecksum(this, 'checksum-{{ file.id }}')">[&nbsp;<a href="#" onclick="return false;">Checksums</a>&nbsp;]</div>
                     <div class="checksums" id="checksum-{{ file.id }}">

_posts/2008-10-19-new-openttd-useful-zip.html

@@ -3,7 +3,7 @@
 author: Rubidium
 ---
 
-Today we created a new <a href="https://www.staging.openttd.org/downloads/openttd-useful/latest.html">openttd-useful.zip</a> with updated libaries and we added a library needed for properly rendering right-to-left scripts.<br />
+Today we created a new <a href="https://www.openttd.org/downloads/openttd-useful-releases/latest.html">openttd-useful.zip</a> with updated libaries and we added a library needed for properly rendering right-to-left scripts.<br />
 If you are compiling yourself on Windows with Microsoft's C compiler from Visual Studio you need to update when you compile trunk or any development branch of OpenTTD as soon as possible, otherwise OpenTTD will fail to compile. For stable releases until the 0.7 series you can still use the old openttd-useful.zip.<br /><br />
 
 If you do not compile OpenTTD yourself or you are not using Microsoft's compiler you will not be affected.

_posts/2009-03-06-openttd-useful-update.html

@@ -3,5 +3,5 @@
 author: Rubidium
 ---
 
-Today we updated <a href="https://www.staging.openttd.org/downloads/openttd-useful/latest.html">openttd-useful.zip</a> as some vulnerabilities were fixed in libpng.<br />
+Today we updated <a href="https://www.openttd.org/downloads/openttd-useful-releases/latest.html">openttd-useful.zip</a> as some vulnerabilities were fixed in libpng.<br />
 If you do not compile OpenTTD yourself or you are not using Microsoft's compiler you will not be affected.

_posts/2009-07-02-openttd-useful-2-3.html

@@ -3,5 +3,5 @@
 author: Rubidium
 ---
 
-Today we updated <a href="https://www.staging.openttd.org/downloads/openttd-useful/latest.html">openttd-useful.zip</a> as some vulnerabilities were fixed in libpng.<br />
+Today we updated <a href="https://www.openttd.org/downloads/openttd-useful-releases/latest.html">openttd-useful.zip</a> as some vulnerabilities were fixed in libpng.<br />
 If you do not compile OpenTTD yourself or you are not using Microsoft's compiler you will not be affected.

_posts/2009-10-16-openttd-useful-2-4.html

@@ -3,5 +3,5 @@
 author: Rubidium
 ---
 
-Today we updated <a href="https://www.staging.openttd.org/downloads/openttd-useful/latest.html">openttd-useful.zip</a> to fix some linking issues with the nightlies/trunk. We also updated the freetype and png libraries.<br />
+Today we updated <a href="https://www.openttd.org/downloads/openttd-useful-releases/latest.html">openttd-useful.zip</a> to fix some linking issues with the nightlies/trunk. We also updated the freetype and png libraries.<br />
 If you do not compile OpenTTD yourself or you are not using Microsoft's compiler you will not be affected.

_posts/2010-01-10-openttd-useful-3-0.html

@@ -3,5 +3,5 @@
 author: Rubidium
 ---
 
-Today we updated <a href="https://www.staging.openttd.org/downloads/openttd-useful/latest.html">openttd-useful.zip</a> to add an extra library. If you want to compile upcoming releases/nightlies/trunk you have to update.<br />
+Today we updated <a href="https://www.openttd.org/downloads/openttd-useful-releases/latest.html">openttd-useful.zip</a> to add an extra library. If you want to compile upcoming releases/nightlies/trunk you have to update.<br />
 If you do not compile OpenTTD yourself or you are not using Microsoft's compiler you will not be affected.

_posts/2010-11-27-openttd-useful-4-0.html

@@ -3,7 +3,7 @@
 author: Rubidium
 ---
 
-Today we updated <a href="https://www.staging.openttd.org/downloads/openttd-useful/latest.html">openttd-useful.zip</a> to change a library to support natural sorting of strings. 
-<br/>If you want to compile upcoming releases/nightlies/trunk you have to update, however if you want to compile OpenTTD before r21340 you will need <a href="https://www.staging.openttd.org/downloads/openttd-useful/latest.html">version 3.4</a>. To be able to compile both OpenTTD revisions before and after r21340 you have to install <a href="https://www.staging.openttd.org/downloads/openttd-useful/latest.html">version 3.4</a> first and then install <a href="https://www.staging.openttd.org/downloads/openttd-useful/latest.html">version 4.0</a> over that.
+Today we updated <a href="https://www.openttd.org/downloads/openttd-useful-releases/latest.html">openttd-useful.zip</a> to change a library to support natural sorting of strings. 
+<br/>If you want to compile upcoming releases/nightlies/trunk you have to update, however if you want to compile OpenTTD before r21340 you will need <a href="https://www.openttd.org/downloads/openttd-useful-releases/latest.html">version 3.4</a>. To be able to compile both OpenTTD revisions before and after r21340 you have to install <a href="https://www.staging.openttd.org/downloads/openttd-useful-releases/latest.html">version 3.4</a> first and then install <a href="https://www.staging.openttd.org/downloads/openttd-useful-releases/latest.html">version 4.0</a> over that.
 <br/><br/>
 If you do not compile OpenTTD yourself or you are not using Microsoft's compiler you will not be affected.

_posts/2010-12-24-openttd-1-1-0-beta1.html

@@ -8,7 +8,7 @@
 include extended support for NewGRFs (especially NewObjects), an industry chain preview and much improved
 handling in the GUI when loading games and configuring NewGRFs (including their parameters). There is also 
 a new admin-interface which allows server control over another network port. Check out the whole list 
-of changes in the <a href="http://binaries.openttd.org/releases/1.1.0-beta1/changelog.txt">changelog</a>.
+of changes in the <a href="https://cdn.openttd.org/openttd-releases/1.1.0-beta1/changelog.txt">changelog</a>.
 </p><p>
 Also we recently started to provide binaries for Mac OS X again, thus this release is also
 available for Mac users.

_posts/2011-01-21-openttd-1-1-0-beta4.html

@@ -9,7 +9,7 @@
 Apart from that, important changes are in the Place Object and Build Rail Station windows. Now, you can select the item you want from a graphical preview - see <a href="http://www.tt-forums.net/viewtopic.php?f=33&t=41222">the forum</a>.<br/>
 Many players and NewGRF developers will welcome better support for NewGRF railtypes, and server admins can now (re)set the password of any company.
 <br/><br/>
-List of important changes is in the <a href="http://binaries.openttd.org/releases/1.1.0-beta4/changelog.txt">changelog</a> - it's even longer than the changelog for beta2 and beta3.
+List of important changes is in the <a href="https://cdn.openttd.org/openttd-releases/1.1.0-beta4/changelog.txt">changelog</a> - it's even longer than the changelog for beta2 and beta3.
 <br/><br/>
 Any bugs you find should be reported at <a href="http://bugs.openttd.org/">our bug tracker</a> - please report them, as it makes OpenTTD better :-)
 <br/><br/>

_posts/2011-02-04-openttd-1-1-0-beta5.html

@@ -5,7 +5,7 @@
 
 <p>Time for yet another beta release in the 1.1.0 series. Quite some bugs have been fixed since the
 last beta and a few further additions have been made and the advanced settings saw some major
-cleanup. You can view the <a href="http://binaries.openttd.org/releases/1.1.0-beta5/changelog.txt">complete changelog</a> at the usual location.</p>
+cleanup. You can view the <a href="https://cdn.openttd.org/openttd-releases/1.1.0-beta5/changelog.txt">complete changelog</a> at the usual location.</p>
 <p>Please report any bugs you might find in this version in <a href="http://bugs.openttd.org/">our bug tracker</a> so that OpenTTD will
 become even better and more stable for the upcoming stable release.
 </p><p>

_posts/2011-02-18-openttd-1-1-0-rc1.html

@@ -3,7 +3,7 @@
 author: planetmaker
 ---
 
-<p>Let us present: the first release candidate for the upcoming OpenTTD 1.1.0. Next to a few new features like a minimal industry density setting or a much improved UI of the console by allowing mouse scrolling there, a well-sized list of bugs has been addressed since the last beta release, among them a number of bugs which crashed OpenTTD under certain conditions. See the full changelog <a href="http://binaries.openttd.org/releases/1.1.0-RC1/changelog.txt">here</a>.
+<p>Let us present: the first release candidate for the upcoming OpenTTD 1.1.0. Next to a few new features like a minimal industry density setting or a much improved UI of the console by allowing mouse scrolling there, a well-sized list of bugs has been addressed since the last beta release, among them a number of bugs which crashed OpenTTD under certain conditions. See the full changelog <a href="https://cdn.openttd.org/openttd-releases/1.1.0-RC1/changelog.txt">here</a>.
 </p><p>
 Please test this <a href="http://www.openttd.org/en/download-testing">release candidate</a> and report any bugs and problems you encounter with it in our <a href="http://bugs.openttd.org">bug tracker</a> in order to help us improve the game further and make sure the final release for 1.1.0 will even be better and more stable.
 </p><p>

_posts/2011-12-24-openttd-1-2-0-beta1.html

@@ -28,7 +28,7 @@
 creation or you can opt to enable maintenance costs for your property which
 allows to counter a bit the linear money increase as costs for property
 increase more than linear with the amount of property which needs maintenance.
-For full list of changes see our <a href=http://binaries.openttd.org/releases/1.2.0-beta1/changelog.txt>changelog</a>.
+For full list of changes see our <a href=https://cdn.openttd.org/openttd-releases/1.2.0-beta1/changelog.txt>changelog</a>.
 </p><p>
 As this is a testing release, we like to ask you to give it a thorough test,
 both new features and 'old stuff' as well. Please help us in making the final

_posts/2012-01-07-openttd-1-2-0-beta2.html

@@ -10,7 +10,7 @@
 But as it's a beta release we still have new features. Among those you can now use the game scripts to ask questions to the companies and check the NewGRF parameters also when playing.
 </p><p>
 Have fun with this new release, give it a proper swirl and test. As usual bugs
-go to our <a href=http://bugs.openttd.org>bug tracker</a>, a full changelog is also <a href=http://binaries.openttd.org/releases/1.2.0-beta2/changelog.txt>available</a> and downloads can be found on our <a href=https://www.openttd.org/downloads/openttd-releases/latest.html>main page</a>.
+go to our <a href=http://bugs.openttd.org>bug tracker</a>, a full changelog is also <a href=https://cdn.openttd.org/openttd-releases/1.2.0-beta2/changelog.txt>available</a> and downloads can be found on our <a href=https://www.openttd.org/downloads/openttd-releases/latest.html>main page</a>.
 </p><p>
 Finally let me remind you of the <a href=http://devs.openttd.org/~planetmaker/titlegame>titlegame competition</a>. Playing and thus creating a nice titlegame might also be a good test ;-)
 </p><p>

_posts/2012-02-19-openttd-1-2-0-rc1.html

@@ -14,7 +14,7 @@
 ahead :-) They might want to visit the <a href=http://newgrf-specs.tt-wiki.net>NewGRF specs wiki</a> for details.</p><p>
 Of course there's also the fair amount of bug fixes, among them handling of refit for vehicles in some cases, 
 networking issues or adjustment of the infrastructure cost for rail which now sums over all rails irrespective of railtype.
-See the <a href=http://binaries.openttd.org/releases/1.2.0-RC1/changelog.txt>full changelog</a> for more
+See the <a href=https://cdn.openttd.org/openttd-releases/1.2.0-RC1/changelog.txt>full changelog</a> for more
 </p><p>
 With this new release we'd like to draw your attention also to the ongoing <a href=http://www.tt-forums.net/viewtopic.php?f=29&t=57555>titlegame competition</a> for this
 new OpenTTD 1.2 branch. Please look <a href=http://devs.openttd.org/~planetmaker/titlegame/round1/>through the entries</a>. For the voting round one please send a ranked list