Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

finger.openttd.org #48

Open
planetmaker opened this Issue Jan 28, 2019 · 12 comments

Comments

Projects
None yet
6 participants
@planetmaker
Copy link
Contributor

planetmaker commented Jan 28, 2019

https://finger.openttd.org/versions.txt lists the 'latest' version for all products OpenTTD offers for download. It currently is not updated. That makes nightly servers virtually impossible.

The other stuff on https://finger.openttd.org/ might be nice to keep updated automatically, too

@TrueBrain

This comment has been minimized.

Copy link
Member

TrueBrain commented Jan 28, 2019

Finger always has been a bit of an issue. And especially with things like kubernetes (on which OpenTTD now runs), it is really difficult to do. I have been wondering about how to solve this.

For now, I think the easiest is to have a "finger" per release type. You can use this file for that:
https://openttd.ams3.digitaloceanspaces.com/openttd-nightlies/listing.txt

It will always show the latest release available; it is in CSV format, with date and name of the release. It bypasses the CDN, meaning you get the most up-to-date version (and not a potentially cached version).

The "branches" and "tags" on finger are completely superseded by GitHub. GitHub allows to fetch this information via their API, and is much more likely to be future-proof. So I would suggest we remove those completely from finger.

Not all downloads are migrated to the new CDN yet, which makes a good solution for finger a bit more difficult. But it was more important to get the new nightlies available than it was to get everything perfect. If there are suggestions what a better replacement for finger is, I would love to hear it. Is an URL like above sufficient? Or should we publish every nightly on GitHub? Or tag every nightly?

Feel free to leave any ideas!

@TrueBrain

This comment has been minimized.

Copy link
Member

TrueBrain commented Feb 1, 2019

Also, not less important, the URL to download binaries from has changed between the old system and new (from https://binaries.openttd.org to https://proxy.binaries.openttd.org). This URL might change again in the future. Not sure how to handle this cleanly. I think that is not really possible.

Biggest hurdle is that the ingame client uses https://binaries.openttd.org for ingame content, and it is unlikely we can deprecate that any time soon (because BaNaNaS is being rewritten, which tights into this). Or I have to find a way to automatically update the CDN with content of BaNaNaS.

Many possibilities, but most of all, tons of annoying issues :D

Either way, still thinking we should stop using "finger" in the current form, and go with the "listing.txt" per release type.

Still looking for additional input!

@claman

This comment has been minimized.

Copy link
Contributor

claman commented Feb 1, 2019

Given that listing.txt will be up-to-date, and that it already exists, that does seem like a good direction to work in.

What's the timeline like on the BaNaNas rewrite? Depending on the projected timing, it might just be easier to wait until that's done.

@Eddi-z

This comment has been minimized.

Copy link
Collaborator

Eddi-z commented Feb 1, 2019

I don't think there is such a thing as a timeline around these parts...

@planetmaker

This comment has been minimized.

Copy link
Contributor Author

planetmaker commented Feb 1, 2019

Ok, I don't mind so much when the download URL and the place to get version info changes. That's about a one-time update to scripts and fine.

However:
AccessDeniedopenttdtx00000000000002638e543-005c54aea9-d7f20f-ams3ad7f20f-ams3a-ams3
is my current result for proxy.binaries.openttd.org

listing.txt as given by you lists the revisions... but what URL could I give a script to get the URL from?

https://www.openttd.org/downloads/openttd-nightlies/20190118-master
https://www.openttd.org/downloads/openttd-nightlies/20190118-master-g07a40df9
both don't work. Nor any link I tried to construct with proxy.b.o.o or similar. What do I miss or not understand?

@TrueBrain

This comment has been minimized.

Copy link
Member

TrueBrain commented Feb 1, 2019

Navigate to https://www.openttd.org, hit the Download nightly button, find the file you are looking for, and there you go :)

https://proxy.binaries.openttd.org/openttd-nightlies/20190201-master-g5a5861f2/openttd-20190201-master-g5a5861f2-source.tar.xz

That is the URL where you can find the source, for example!

If you want the direct URL on the webpage:
https://www.openttd.org/downloads/openttd-nightlies/20190201-master-g5a5861f2.html

But I am guessing you want the download URL more than anything else.

Not sure why you are trying the 20190118 version; that is rather old :D

Not sure why I mentioned listing.txt; I think latest.txt is more useful for you:

https://openttd.ams3.digitaloceanspaces.com/openttd-nightlies/latest.txt

Hope it helps!

@TrueBrain

This comment has been minimized.

Copy link
Member

TrueBrain commented Feb 9, 2019

With the addition of releasees, it turns out that latest.txt is not sensible anymore. Is latest the testing or stable release? So I added to the listing.txt a new field name, which for example is always master for nightlies, but testing or stable for releases. And I removed latest.txt.

Please let me know if you are scripting things with this, and see other possibilities :)

https://openttd.ams3.digitaloceanspaces.com/openttd-nightlies/listing.txt
https://openttd.ams3.digitaloceanspaces.com/openttd-releases/listing.txt
https://openttd.ams3.digitaloceanspaces.com/openttd-pullrequests/listing.txt

@andreszs

This comment has been minimized.

Copy link

andreszs commented Feb 14, 2019

Hi, I reported the issue here and also suggested the implementation of meta tags.

The https://openttd.ams3.digitaloceanspaces.com/openttd-releases/listing.txt does not include stable releases, is this intentional? Where can we get info about latest stable release? Also, implementing a single XML file would be more clean than this multiple TXT files method...

Also, what's with the bizarre "openttd.ams3.digitaloceanspaces.com" URL? Where does it come from and how can we ensure this URL won't change in the near future? I think it would be more reliable to request them from the http://www.openttd.org server directly... Thanks.

@TrueBrain

This comment has been minimized.

Copy link
Member

TrueBrain commented Feb 16, 2019

Let me see if I get all the questions here ... (forgive me if I miss any).

Please create a new issue about 'meta tags', or even better, a Pull Request ;) I have yet to look into it, what the best practices are, etc. Tnx!

I will see what I can do about the URL itself. Currently it just points to the CDN, but as it doesn't support IPv6, having it wrapped in some openttd.org URL might be nice. You could use proxy.binaries.openttd.org, but those files are cached for at least a day on the edge. That is not ideal for these kind of files. So we will have to cook up some other solution. I will put that on my todo. Good suggestion.

Currently these URLs only list what has been published on the new infrastructure. So it is not listing any stable, as there are no stables on the new infrastructure. Currently, a combination of 'finger' and these URLs are needed for the complete overview (as where the files are located also differ between those two). Soon that will be fixed if we release our first stable (and finger can be deprecated completely). Or I might manually migrate an older stable to the new infrastructure. (depends a bit on the timeline for the next stable, I guess).

We won't be publishing any XML files anymore. XML is a very unfriendly format, both for machine and human. I considered YAML briefly for the listing.txt, but these text-files are easier to process by human and by script (both from reading as from writing point of view). I currently see no need to change the format (but, if presented a use case, this is something we can talk about). As it goes with these suggestions, as you present only a solution, I have no clue what the question is behind the solution, so I am a bit guessing here :) Feel free to show what you are trying to solve, and we can chat about it!

We won't be publishing a single file anymore with everything in there. Currently how 'finger' works is unmaintainable, especially as we want to allow building and publishing things like patchpacks etc. Currently I think it is sufficient to have these listing.txt files, and so far people I have talked with have had no issues. If you can explain the usecase you are facing making you ask that question, we can have a chat about it. (again here, you present a solution; I would love to understand the problem you try to solve, so we can think with you in what the best approach is).

There was a lot to unfold in your comment; I hope I have them all :) Lets please continue the talk, and see what we can do to make everyone happy here! This is all about collaboration :) Tnx!

@TrueBrain

This comment has been minimized.

Copy link
Member

TrueBrain commented Feb 16, 2019

Another suggestion that came up, that I am going to leave here, is to create a REST API which publishes this information. That would mean making this compatible etc might be a bit easier.

This just doesn't exist at all yet, and has to be created from scratch.

@andreszs

This comment has been minimized.

Copy link

andreszs commented Feb 21, 2019

Sorry for the delay, the idea behind the XML is to download a single file (faster and more reliable than multiple HTTP queries) in order to populate the info required for my OpenTTD Manager to show versions info, and allow direct download:

image

As you can see, all I need is version number, date, and download URL. I suggested XML because VB6 features very simple XML parser.

I strongly disagree with the comment about XML being an unfriendly format. Please check out the following sample XML that would be needed in this case, and let me know if you find it complicated. XML does become cumbersome and tedious when additional features such as namespaces or attributes are introduced, they could easily drive mad any developer / parser, but they are not required here. As a personal opinion, the guy who invented namespaces should be lapidated.

<?xml version="1.0" encoding="UTF-8"?>
<stable>
	<version>1.8.0</version>
	<date>2018-04-01 13:07 UTC</date>
	<download>
		<win9x>https://binaries.openttd.org/releases/1.8.0/openttd-1.8.0-windows-win9x.zip</win9x>
		<win32>https://binaries.openttd.org/releases/1.8.0/openttd-1.8.0-windows-win32.zip</win32>
		<win64>https://binaries.openttd.org/releases/1.8.0/openttd-1.8.0-windows-win64.zip</win64>
	</download>
</stable>
<testing>
	<version>1.9.0-beta2</version>
	<date>2019-02-09 21:41 UTC</date>
	<download>
		<win32>https://proxy.binaries.openttd.org/openttd-releases/1.9.0-beta2/openttd-1.9.0-beta2-windows-win32.zip</win32>
		<win64>https://proxy.binaries.openttd.org/openttd-releases/1.9.0-beta2/openttd-1.9.0-beta2-windows-win64.zip</win64>
	</download>
</testing>
<master>
	<version>20190220-master-g21f009dc78</version>
	<date>2019-02-20 19:01 UTC</date>
	<download>
		<win32>https://proxy.binaries.openttd.org/openttd-nightlies/20190220-master-g21f009dc78/openttd-20190220-master-g21f009dc78-windows-win32.zip</win32>
		<win64>https://proxy.binaries.openttd.org/openttd-nightlies/20190220-master-g21f009dc78/openttd-20190220-master-g21f009dc78-windows-win64.zip</win64>
	</download>
</master>

The ill-use of XML may have damaged its reputation, but the XML format at its core remains simple and elegant. The currently used text files could change and are not very useful, specially when their URL keeps changing and no one knows what info and in what order could they provide. XML avoids all these problems, and allows adding info without breaking current implementations.

The Previous Versions combo box is populated from some binaries URL, but this might be removed because that URL is not updated anymore either.

@Leffe108

This comment has been minimized.

Copy link

Leffe108 commented Mar 15, 2019

Just joining in a bit. I'm Zuu on the forums/irc and ran OpenTTD Auto Update for some years, but I do not plan to maintain it further. If I can say something, it is to if possible publish information such that it allow standard tools on platforms to use it. That being said I don't know any standardized tool on Windows to upload software using some sort of finger server.

For the big releases it may be possible to setup CI/CD to push to windows store. If it would be feasible to have a nightly channel there I don't know. If windows store is not possible, then maybe there is something like apt but similar for programs on windows we could work with that is open and reasonable to offer as either you use this or you download on your own or build it yourself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.