Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Testing out new download server - please report any issues here #1498

Closed
justinclift opened this issue Aug 9, 2018 · 20 comments

Comments

Projects
None yet
4 participants
@justinclift
Copy link
Member

commented Aug 9, 2018

Just a heads up, in a few minutes I'll update (just) the DB4S 3.10.1 release binary for Win32 to point at the new in-development download cluster:

    https://download.sqlitebrowser.org/DB.Browser.for.SQLite-3.10.1-win32.exe

This is an experiment / test, just to start getting an idea of what it would take for us to serve downloads and the currentrelease file from our own infrastructure.

Please report and all weirdness with the downloads here, so it can be investigated / rolled-back / etc as needed. 😄

@chrisjlocke

This comment has been minimized.

Copy link
Contributor

commented Aug 9, 2018

Downloaded it fine here - no problems. Didn't install it, so haven't tested the 'currentrelease' version file... yell back if you want me to...

@justinclift

This comment has been minimized.

Copy link
Member Author

commented Aug 9, 2018

Thanks @chrisjlocke. I've been looking at other things, and haven't yet set up proper monitoring... but in general when logging into the 2 (very small) servers they're barely feeling any load at all. Almost no disk nor memory io, cpu is around 5% too.

I need to find better tooling for measuring bandwidth though, as although the bandwidth isn't charged for (real "unlimited" style) each server can only do 200Mbit/s. We need to know how much bandwidth we use during peak download times for new releases, but at least this is a start to working things out. 😄

For the "currentrelease" file, there's nothing much really to test yet. It'll just serve up the same file contents that's currently in our GitHub repo

Later on today I'll probably update the DB4S git repo to use this new server for the currentrelease file, which means for subsequent nightly builds they'll start to use it. Then we can see what bug reports turn up. 😉

@jungle-boogie

This comment has been minimized.

Copy link
Contributor

commented Aug 9, 2018

In the past, I've used vnstat:
https://humdi.net/vnstat

@justinclift

This comment has been minimized.

Copy link
Member Author

commented Aug 9, 2018

Thanks @jungle-boogie, haven't tried that one before. It's in the CentOS 7 repo's, so I'll give it a go now.

I'll probably set up a Zabbix or Prometheus box for "real monitoring" in near-ish future too. 😄

@justinclift justinclift added the website label Aug 9, 2018

@justinclift

This comment has been minimized.

Copy link
Member Author

commented Aug 9, 2018

That vmstat is turning out to be useful already, as it seems like a good light weight solution to tracking historical bandwidth usage without needing a full on solution.

For the past few hours we've been transmitting about 1/2 a GB per hour for the Win32 download.

I'll add the Win64 one to it in a minute, which should triple that. Let's see... 😄

@justinclift

This comment has been minimized.

Copy link
Member Author

commented Aug 9, 2018

Hmmm, this new approach seems to have defaulted to IPv4 only. I'll revert it for now, and take this up again tomorrow when I'm more awake. 😄

@justinclift

This comment has been minimized.

Copy link
Member Author

commented Aug 10, 2018

Well, I've managed to fix the problem with client IP addresses being masked. So that's a win.

On the other hand, the current level of IPv6 support in Docker does not seem fit for purpose:

    https://docs.docker.com/compose/compose-file/#enable_ipv6

"Swarm" mode being the orchestration approach that's integrated with Docker. I can really see why people say Docker is useful for development, but just makes things harder for production.

Containerising any kind of "Business Critical" application seems like it would be fraught with danger (and very unwise). The Docker networking code seems super unstable. Lots and lots of bug reports about regressions, general breakage, etc. And it seems to have been an ongoing problem for years. 😲

Maybe I just need to sleep on this and look again tomorrow. 😄

@justinclift

This comment has been minimized.

Copy link
Member Author

commented Aug 10, 2018

Tomorrow, I'll probably just un-dockerise this, as it's adding a layer of crappiness and it's lack of native IPv6 support just leads me to consider Docker "not fit for purpose" for production workloads of small teams. At least, not those requiring IPv6 support.

If we were a large business with dedicated IT support, then maybe it'd suit. In that scenario we'd have people on staff that could likely implement and maintain a full k8s+Calico stack to put this stuff in. eg working IPv6 in containers.

That's not where we're at though. 😉

@justinclift

This comment has been minimized.

Copy link
Member Author

commented Aug 10, 2018

Looks like Kubernetes is getting around to adding IPv6 support too, without the need to use a 3rd party network component like Calico.

Likely way too much admin overhead for our simple use case.

@justinclift

This comment has been minimized.

Copy link
Member Author

commented Aug 10, 2018

Ahhh, if we downgrade to Docker CE 17.12, we should be able to get IPv6 support happening. It looks like there's two streams of development happening presently, and the v18.x releases have "Swarm mode" but no IPv6 support.

I wonder if Docker CE 17.12 on CentOS is still considered "supported"? eg receives security fixes.

I'll investigate that tomorrow, and if it is then we can keep using Docker. If it's not, then I'll un-dockerise the new download boxes, as IPv6 support (for us) is mandatory for our public facing services.

@justinclift

This comment has been minimized.

Copy link
Member Author

commented Aug 11, 2018

k, the new download bits are running directly on the hosts now (no longer in Docker), so are listening on both IPv4 and IPv6. I've just updated the main DB4S website to point all four v3.10.1 downloads at them, so lets see how it goes this time. 😄

@MKleusberg

This comment has been minimized.

Copy link
Member

commented Aug 11, 2018

Heh heh, "Docker does not seem fit for purpose" and "it's adding a layer of crappiness". I never really understood the hype 😉

@justinclift

This comment has been minimized.

Copy link
Member Author

commented Aug 11, 2018

Well, it does seem to work pretty well for "internal" machines, including when they need to scale from a single server to small cluster size. It's very much a "set it up, and it just seems to run". Including self-healing for services (with the right parameters in place).

Troubleshooting on the other hand, seem like it can be a real PITA.

For untrusted environments though (eg internet facing), and things needing IPv6... it's definitely not there yet. As in "very much avoid" unless there are dedicated people handy with the specific (non-default) technologies needed.

If the IPv6 stack and other issues I hit on the way through do get smoothed out over the next (say) 18-24 months, then I reckon it'll have really good uptake from there. The CLI is also pretty rough. It reminds me of the Libvirt CLI before I got involved with that and started knocking off the rough edges. 😄

@justinclift

This comment has been minimized.

Copy link
Member Author

commented Aug 13, 2018

Today's our first weekday using the new download servers for all 4 of the 3.10.1 downloads (as linked from the main website). There's only 2 servers online atm, and they're set up in round robin fashion... so super low effort approach for spreading load.

It looks like each download server will have generated about 30-40GB of traffic (mostly sending of course) for the full day. It's not a full day yet, so that's an estimate (from vnstat).

If that's consistent (eg "steady" state), these two servers should comfortably handle the load we're seeing. From memory of our GitHub page view stats, our traffic there generally doubles for a week or two after new releases.

So, for our next release we might want to bring an extra server or two online to handle the peak period, then drop it back to just these two after things settle down. Guessing (!) that would continue to present a fairly ok download experience for people without notice-able slowness of downloads.

Of course, with this being round-robin (no load balancer in front), things could definitely go pear shaped if any particular server happens to cop a large non-shared load. That's not a problem for today though (hopefully). 😁

@justinclift

This comment has been minimized.

Copy link
Member Author

commented Aug 18, 2018

The new download servers seem to be working fine, and there haven't been any complaints about moving the "currentrelease" file to the new servers. The new servers are definitely listening on both IPv4 and IPv6 too (seeing source addresses for both come in), so this looks to be a win already. 😄

@jungle-boogie

This comment has been minimized.

Copy link
Contributor

commented Sep 24, 2018

The new download location doesn't seem to have the two separate builds of sqlitebrowser, one built with sqlite and one with the cipher thing. Only the latter is available. Is that deliberate?

https://download.sqlitebrowser.org/

@jungle-boogie

This comment has been minimized.

Copy link
Contributor

commented Sep 24, 2018

To be more clear....
https://nightlies.sqlitebrowser.org/latest/

DB.Browser.for.SQLite-win64.msi doesn't have the cipher thing built with it, but has the latest sqlite build (at the time of the last modify day, anyway)

DB.Browser.for.SQLite-sqlcipher-win64.msi has sqlcipher thing

The new download path has only the sqlcipher thing:
https://download.sqlitebrowser.org/

@justinclift

This comment has been minimized.

Copy link
Member Author

commented Sep 24, 2018

Yeah, that's deliberate. The new download servers only have the last release (for now), which only has a SQLCipher enabled version. 😄

@jungle-boogie

This comment has been minimized.

Copy link
Contributor

commented Sep 24, 2018

@justinclift

This comment has been minimized.

Copy link
Member Author

commented Sep 24, 2018

Not sure atm. Pretty likely it'll be something kind of old now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.