New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RawPedia (HostGator) is down #3550

Closed
Vittel opened this Issue Dec 20, 2016 · 19 comments

Comments

Projects
None yet
7 participants
@Vittel

Vittel commented Dec 20, 2016

If you are filing a bug report, please first read the guide How to Write Useful Bug Reports:
http://rawpedia.rawtherapee.com/How_to_write_useful_bug_reports
You must provide the required information in order for your bug report to be valid.
Delete this text before posting.

I WOULD LOVE TO DO SO, BUT: http://rawpedia.rawtherapee.com/ -> 404 -> sad

=)

@Beep6581

This comment has been minimized.

Show comment
Hide comment
@Beep6581

Beep6581 Dec 20, 2016

Owner

Oh boy!

Owner

Beep6581 commented Dec 20, 2016

Oh boy!

@Beep6581 Beep6581 self-assigned this Dec 20, 2016

@Beep6581

This comment has been minimized.

Show comment
Hide comment
@Beep6581

Beep6581 Dec 20, 2016

Owner

Status update: the files are intact, I presume the database is intact as well. HostGator blocked connecting via IP. Looking into solutions.

Owner

Beep6581 commented Dec 20, 2016

Status update: the files are intact, I presume the database is intact as well. HostGator blocked connecting via IP. Looking into solutions.

@Beep6581 Beep6581 changed the title from Documentation is down! to RawPedia (HostGator) is down Dec 20, 2016

@Beep6581 Beep6581 added the website label Dec 20, 2016

@patdavid

This comment has been minimized.

Show comment
Hide comment
@patdavid

patdavid Dec 21, 2016

Is there something I can do?

patdavid commented Dec 21, 2016

Is there something I can do?

@Beep6581

This comment has been minimized.

Show comment
Hide comment
@Beep6581

Beep6581 Dec 21, 2016

Owner

Status update: HostGator requires that we use a domain to connect to the server. Our Hungarian server expires in July 2017, I intended to make a new website by then and move it to HostGator and point our rawtherapee.com domain to it. Now our arm is twisted and that migration needs to be speeded up.

Options:

  1. Keep using HostGator. I have not had issues with them ever until this surprise.
    • Pros: Gabor paid for another 3 years of hosting with them just two weeks ago.
    • Cons: its the cheapest shared hosting option they offer (hatchling plan at $8.80/month) (MediaWiki (RawPedia runs on MediaWiki) ran slowly on it, though a normal website runs fine. I now have SSH access to it, so there is a possibility that if I re-install MediaWiki I can get a proper cache up and running which will make MediaWiki run faster.
  2. Keep using HostGator but upgrade either to cloud hosting (~10$/month) or to VPS (20$/month).
    • Pros: VPS would be best,
    • Cons: but paying that much would also be overkill because our needs are low, and the only reason we'd need VPS is to make MediaWiki run faster. Maybe cloud would be enough, but I don't know where exactly the bottleneck in MediaWiki's slow speed is so can't say for sure.
  3. Use a different hosting company - DigitalOcean or other.
  4. We already share a forum, maybe we can share some server space. If we had VPS we'd be paying for much more than we use. Maybe someone else is also using less than they have and would be willing to co-exist, @patdavid ?
    • Pros: Share costs, fast server, we both win. Share expertise too.
    • Cons: Sharing the server would require that we can hook up our rawtherapee.com domain to our part of the server, I've not done that before.

Our requirements are very low. The new website will be static. Only MediaWiki will require some access to the server to enable proper caching so that it runs fast.

Owner

Beep6581 commented Dec 21, 2016

Status update: HostGator requires that we use a domain to connect to the server. Our Hungarian server expires in July 2017, I intended to make a new website by then and move it to HostGator and point our rawtherapee.com domain to it. Now our arm is twisted and that migration needs to be speeded up.

Options:

  1. Keep using HostGator. I have not had issues with them ever until this surprise.
    • Pros: Gabor paid for another 3 years of hosting with them just two weeks ago.
    • Cons: its the cheapest shared hosting option they offer (hatchling plan at $8.80/month) (MediaWiki (RawPedia runs on MediaWiki) ran slowly on it, though a normal website runs fine. I now have SSH access to it, so there is a possibility that if I re-install MediaWiki I can get a proper cache up and running which will make MediaWiki run faster.
  2. Keep using HostGator but upgrade either to cloud hosting (~10$/month) or to VPS (20$/month).
    • Pros: VPS would be best,
    • Cons: but paying that much would also be overkill because our needs are low, and the only reason we'd need VPS is to make MediaWiki run faster. Maybe cloud would be enough, but I don't know where exactly the bottleneck in MediaWiki's slow speed is so can't say for sure.
  3. Use a different hosting company - DigitalOcean or other.
  4. We already share a forum, maybe we can share some server space. If we had VPS we'd be paying for much more than we use. Maybe someone else is also using less than they have and would be willing to co-exist, @patdavid ?
    • Pros: Share costs, fast server, we both win. Share expertise too.
    • Cons: Sharing the server would require that we can hook up our rawtherapee.com domain to our part of the server, I've not done that before.

Our requirements are very low. The new website will be static. Only MediaWiki will require some access to the server to enable proper caching so that it runs fast.

@patdavid

This comment has been minimized.

Show comment
Hide comment
@patdavid

patdavid Dec 21, 2016

@Beep6581

At the moment there are a couple of options if you'd like to pursue them...

The infrastructure for pixls is on two separate servers:

  • Stablehost for a shared webhosting plan (unlimited).
    This has an option for installing mediawiki, but I have no idea how much I can tune a cache for performance - probably worth checking out in greater detail. I'll poke around later if possible.
  • Digital Ocean (2GB ram, 2 cores, 40GB SSD) for the forum.
    We could set up another virtual host here. It's a full VPS, so we can certainly install whatever is needed.

We can easily migrate the main website to the shared webserver on Stablehost. If it's fully static we might be able to let Travis build it and push it (similar to how the main pixls website works now). Or we can move it onto a virtual host on the DO VPS machine as well. Low impact and relatively simple to do, I think.

Do we know what type of traffic we are seeing and server load for rawpedia? If it's not too bad we might be able to manage it on the existing DO VPS with the forums.

Otherwise, we can look at bumping the servers to accommodate, which I am ok with of course.

Options:

  • Bump DO droplet to 4GB, 2 cores, 60GB ssd ($40/mo).
  • Get new VPS at Stablehost 4GB ram, 4 cores, 100GB ssd ($30/mo).
  • We could also look at something like OSU Open Source lab for possible hosting, or approach some other type of patron that might be willing to give us space.

patdavid commented Dec 21, 2016

@Beep6581

At the moment there are a couple of options if you'd like to pursue them...

The infrastructure for pixls is on two separate servers:

  • Stablehost for a shared webhosting plan (unlimited).
    This has an option for installing mediawiki, but I have no idea how much I can tune a cache for performance - probably worth checking out in greater detail. I'll poke around later if possible.
  • Digital Ocean (2GB ram, 2 cores, 40GB SSD) for the forum.
    We could set up another virtual host here. It's a full VPS, so we can certainly install whatever is needed.

We can easily migrate the main website to the shared webserver on Stablehost. If it's fully static we might be able to let Travis build it and push it (similar to how the main pixls website works now). Or we can move it onto a virtual host on the DO VPS machine as well. Low impact and relatively simple to do, I think.

Do we know what type of traffic we are seeing and server load for rawpedia? If it's not too bad we might be able to manage it on the existing DO VPS with the forums.

Otherwise, we can look at bumping the servers to accommodate, which I am ok with of course.

Options:

  • Bump DO droplet to 4GB, 2 cores, 60GB ssd ($40/mo).
  • Get new VPS at Stablehost 4GB ram, 4 cores, 100GB ssd ($30/mo).
  • We could also look at something like OSU Open Source lab for possible hosting, or approach some other type of patron that might be willing to give us space.
@English-Wolf

This comment has been minimized.

Show comment
Hide comment
@English-Wolf

English-Wolf Dec 21, 2016

I am not a programmer, I am a casual user of rawtherapee.

I have an inordinate amount of space and traffic on my ISP (Dreamhost) that I use since over a decade ago. I host a few low traffic domains on a shared IP.

If you are interested let me know and let me know the requirement you have. Cost is $0.00

English-Wolf commented Dec 21, 2016

I am not a programmer, I am a casual user of rawtherapee.

I have an inordinate amount of space and traffic on my ISP (Dreamhost) that I use since over a decade ago. I host a few low traffic domains on a shared IP.

If you are interested let me know and let me know the requirement you have. Cost is $0.00

@kulga

This comment has been minimized.

Show comment
Hide comment
@kulga

kulga Dec 22, 2016

If the requirements are simply a mediawiki and static site, I'd grab a tiny VPS, run nginx + php-fpm.
Then do zend-opcache for php, use nginx builtin caching for anonymous traffic for the mediawiki site.
VPS is about 256MB and $2.6 month ($8/qtr) with ramnode

I'm also happy to help host as well. Been using rawtherapee for several years. Be nice to contribute back. I've been hosting sites for a few other people and this looks easy enough

  • Ramnode VPS in Los Angeles (1Gb, quad core) - Speed Test
  • Free ssl (Let's encrypt)
  • Bi-hour, daily, weekly, monthly local & remote backup to a dedicated backup VPS.
  • Caching at php and webserver levels
  • Several layers of security: custom firewall, separation of privileges, separated php processes, etc.
  • I could wrangle up database and sftp access

kulga commented Dec 22, 2016

If the requirements are simply a mediawiki and static site, I'd grab a tiny VPS, run nginx + php-fpm.
Then do zend-opcache for php, use nginx builtin caching for anonymous traffic for the mediawiki site.
VPS is about 256MB and $2.6 month ($8/qtr) with ramnode

I'm also happy to help host as well. Been using rawtherapee for several years. Be nice to contribute back. I've been hosting sites for a few other people and this looks easy enough

  • Ramnode VPS in Los Angeles (1Gb, quad core) - Speed Test
  • Free ssl (Let's encrypt)
  • Bi-hour, daily, weekly, monthly local & remote backup to a dedicated backup VPS.
  • Caching at php and webserver levels
  • Several layers of security: custom firewall, separation of privileges, separated php processes, etc.
  • I could wrangle up database and sftp access
@michaelezra

This comment has been minimized.

Show comment
Hide comment
@michaelezra

michaelezra Dec 23, 2016

Collaborator
  1. I vote to drop HostGator!!!!:)

  2. rawpedia must be made accessible with correct domain name - not via IP.

  3. I've been using 1and1.com for numerous websites for about 10 years and am satisfied with service and phone support.

1and1.com Linux plans: https://www.1and1.com/web-hosting

I currently have Unlimited Pro account with 2GB RAM, unlimited storage, unlimited bandwidth, unlimited emails with 50GB mailboxes, MySQL, PHP, sFTP, etc. and am using it for few wordpress websites.

I can host RT under my existing account at no extra cost to the project, although I am not entirely sure what performance impact this may have. Otherwise, their pricing is negotiable with 1+ year contracts.

Collaborator

michaelezra commented Dec 23, 2016

  1. I vote to drop HostGator!!!!:)

  2. rawpedia must be made accessible with correct domain name - not via IP.

  3. I've been using 1and1.com for numerous websites for about 10 years and am satisfied with service and phone support.

1and1.com Linux plans: https://www.1and1.com/web-hosting

I currently have Unlimited Pro account with 2GB RAM, unlimited storage, unlimited bandwidth, unlimited emails with 50GB mailboxes, MySQL, PHP, sFTP, etc. and am using it for few wordpress websites.

I can host RT under my existing account at no extra cost to the project, although I am not entirely sure what performance impact this may have. Otherwise, their pricing is negotiable with 1+ year contracts.

@Vittel

This comment has been minimized.

Show comment
Hide comment
@Vittel

Vittel Dec 23, 2016

In the meantime: how about generating a classic pdf help file and put it where the others are?
It would be very helpful.

Vittel commented Dec 23, 2016

In the meantime: how about generating a classic pdf help file and put it where the others are?
It would be very helpful.

@Beep6581

This comment has been minimized.

Show comment
Hide comment
@Beep6581

Beep6581 Dec 23, 2016

Owner

Thank you everyone for the generous help! I am discussing details with Gabor. The situation is that HostGator changed their terms of use without our consent, as a result of which we can no longer continue using their service in its current form. If we cancel the account, the priority is to get him a refund, as he paid a lot of money (several hundred dollars) just two weeks ago or so for a 3 year renewal. We will try to decide today how to proceed.

@michaelezra you have recommended 1and1 before. As you still recommend them, and as you are a project member, I am leaning towards your option if we choose to self-host. But sharing forums with Pixls.us has been a great move from my point of view, and I am heavily considering going with @patdavid 's option, so we can share costs and responsibility, and if someone goes wrong someone will be awake to handle it.

Statistics for rawtherapee.com (not RawPedia):

  • Disk usage: almost nothing if you exclude raw files and builds. Raw files won't be hosted on the new site because they will be hosted in the pixls raw archive. RawTherapee builds take only a few hundred MB and this number will grow very slowly, because we delete old nightly builds when new ones come out, we only keep the actual releases.
  • Average bandwidth of 1500GB per month. This can be cut down, probably drastically, as I believe the majority is bogus. For example I had to cut 7 Chinese IP addresses as they alone were downloading hundreds of GB each.
  • Top bandwidth users:
    • USA 125GB,
    • Germany 76GB,
    • Poland 67GB,
    • China 65GB,
    • France 56GB,
    • Russia 53GB,
    • Italy 39GB
  • Top downloaded file types:
    • Archive 871GB
    • JPG 30GB (this is one of the reasons I think the majority of the bandwidth usage is bogus, only a bot would dig deep enough into our site to download this amount of image data)
    • PDF 17GB
  • Top downloads:
    • /builds/windows/RawTherapee_WinVista_64_Gtk3_Release_4.2.1375.zip (10MB), 17000 hits, 346GB
    • /builds/mac/RawTherapee_OSX_10.8_4.2.1408_gtk3.zip (28MB), 4500 hits, 157GB
  • Operating systems:
    • Windows 62%
    • Linux 12%
    • macOS 10%
  • Web browsers:
    • Chrome 36%
    • Firefox 27%
    • IE 9%
  • Top referrers:

I have no statistics for RawPedia but the bandwidth is very low and the disk usage is under 200MB IIRC.

@patdavid would you be willing to try hosting RawPedia? We can wait with the website until a new one is ready.
https://www.mediawiki.org/wiki/Manual:Caching
https://www.mediawiki.org/wiki/Manual:Performance_tuning
We can discuss details elsewhere if you like.

Owner

Beep6581 commented Dec 23, 2016

Thank you everyone for the generous help! I am discussing details with Gabor. The situation is that HostGator changed their terms of use without our consent, as a result of which we can no longer continue using their service in its current form. If we cancel the account, the priority is to get him a refund, as he paid a lot of money (several hundred dollars) just two weeks ago or so for a 3 year renewal. We will try to decide today how to proceed.

@michaelezra you have recommended 1and1 before. As you still recommend them, and as you are a project member, I am leaning towards your option if we choose to self-host. But sharing forums with Pixls.us has been a great move from my point of view, and I am heavily considering going with @patdavid 's option, so we can share costs and responsibility, and if someone goes wrong someone will be awake to handle it.

Statistics for rawtherapee.com (not RawPedia):

  • Disk usage: almost nothing if you exclude raw files and builds. Raw files won't be hosted on the new site because they will be hosted in the pixls raw archive. RawTherapee builds take only a few hundred MB and this number will grow very slowly, because we delete old nightly builds when new ones come out, we only keep the actual releases.
  • Average bandwidth of 1500GB per month. This can be cut down, probably drastically, as I believe the majority is bogus. For example I had to cut 7 Chinese IP addresses as they alone were downloading hundreds of GB each.
  • Top bandwidth users:
    • USA 125GB,
    • Germany 76GB,
    • Poland 67GB,
    • China 65GB,
    • France 56GB,
    • Russia 53GB,
    • Italy 39GB
  • Top downloaded file types:
    • Archive 871GB
    • JPG 30GB (this is one of the reasons I think the majority of the bandwidth usage is bogus, only a bot would dig deep enough into our site to download this amount of image data)
    • PDF 17GB
  • Top downloads:
    • /builds/windows/RawTherapee_WinVista_64_Gtk3_Release_4.2.1375.zip (10MB), 17000 hits, 346GB
    • /builds/mac/RawTherapee_OSX_10.8_4.2.1408_gtk3.zip (28MB), 4500 hits, 157GB
  • Operating systems:
    • Windows 62%
    • Linux 12%
    • macOS 10%
  • Web browsers:
    • Chrome 36%
    • Firefox 27%
    • IE 9%
  • Top referrers:

I have no statistics for RawPedia but the bandwidth is very low and the disk usage is under 200MB IIRC.

@patdavid would you be willing to try hosting RawPedia? We can wait with the website until a new one is ready.
https://www.mediawiki.org/wiki/Manual:Caching
https://www.mediawiki.org/wiki/Manual:Performance_tuning
We can discuss details elsewhere if you like.

@patdavid

This comment has been minimized.

Show comment
Hide comment
@patdavid

patdavid Dec 23, 2016

@Beep6581 - I've setup MediaWiki 1.28 on pixls and I've run through a few things on the optimization links (opcache, memcached, varnish all up and running). Hit me up on PM at discuss and we can work out getting your backups and restoring them here. If things work out we can fire it all back up soon. :)

patdavid commented Dec 23, 2016

@Beep6581 - I've setup MediaWiki 1.28 on pixls and I've run through a few things on the optimization links (opcache, memcached, varnish all up and running). Hit me up on PM at discuss and we can work out getting your backups and restoring them here. If things work out we can fire it all back up soon. :)

@Beep6581

This comment has been minimized.

Show comment
Hide comment
@Beep6581
Owner

Beep6581 commented Dec 23, 2016

@patdavid great!

@English-Wolf

This comment has been minimized.

Show comment
Hide comment
@English-Wolf

English-Wolf Dec 23, 2016

English-Wolf commented Dec 23, 2016

@kulga

This comment has been minimized.

Show comment
Hide comment
@kulga

kulga Dec 24, 2016

You mention some high bandwidth usage. Yep. You appear to have minimal security there. I am able to fairly reliable download at about 3MB/s from your server. I'm also able to get about 1MB/s from discuss.pixls.us. This is with a single file about 30kb in size. You should not be allowing me to do this! Also, iou about 300MB in bandwidth - sorry 🎱
Three things:
1)

  • Top downloaded file types:
    • Archive 871GB
    • JPG 30GB (this is one of the reasons I think the majority of the bandwidth usage is bogus, only a bot would dig deep enough into our site to download this amount of image data)
    • PDF 17GB

If using nginx, I suggest separating the builds / zips into their own directory and rate limiting their download after a certain amount (eg 10M)
Eg.


location ^~ /builds/ {
limit_rate_after 10m;
limit_rate 500k;
}

... As well as rate limiting image / video files to something reasonable. 500k is generally fine but trial and error is smart. Limit connections as well to something a reasonable user would never trod upon.

Lock down security.
Ftp should really not be used, plain text auth shouldn't be used, but should be ok as long as you have good passwords and login from secure places..
Use iptables (firewall) and limit the number of new connections to your server to something that normal users never reach. This is really important as it makes other things (like bruteforcing a login) much more inconvenient. I've stopped many a DOS this way.
3)
Have a effective backup strategy. Are you only making backups to the local server? to another server?
Can the webuser (through php) modify / delete these backups?
Are the backups automatic and routine? Just files or database and files?
Ideally, I should be able to login to your php user, delete everything I have access to and restoring it should be a inconvenience.

In summary, limit connection speed for some types of files, use a more effective firewall with rate-limiting and have a robust tested backup system.


Apologies for the long post and bandwidth usage :s - system admin / security are hobbies of mine. I sometimes help people who haven't a idea with system admin :) - It's also somewhat off topic so I'll leave it be with this post.
Good luck!

kulga commented Dec 24, 2016

You mention some high bandwidth usage. Yep. You appear to have minimal security there. I am able to fairly reliable download at about 3MB/s from your server. I'm also able to get about 1MB/s from discuss.pixls.us. This is with a single file about 30kb in size. You should not be allowing me to do this! Also, iou about 300MB in bandwidth - sorry 🎱
Three things:
1)

  • Top downloaded file types:
    • Archive 871GB
    • JPG 30GB (this is one of the reasons I think the majority of the bandwidth usage is bogus, only a bot would dig deep enough into our site to download this amount of image data)
    • PDF 17GB

If using nginx, I suggest separating the builds / zips into their own directory and rate limiting their download after a certain amount (eg 10M)
Eg.


location ^~ /builds/ {
limit_rate_after 10m;
limit_rate 500k;
}

... As well as rate limiting image / video files to something reasonable. 500k is generally fine but trial and error is smart. Limit connections as well to something a reasonable user would never trod upon.

Lock down security.
Ftp should really not be used, plain text auth shouldn't be used, but should be ok as long as you have good passwords and login from secure places..
Use iptables (firewall) and limit the number of new connections to your server to something that normal users never reach. This is really important as it makes other things (like bruteforcing a login) much more inconvenient. I've stopped many a DOS this way.
3)
Have a effective backup strategy. Are you only making backups to the local server? to another server?
Can the webuser (through php) modify / delete these backups?
Are the backups automatic and routine? Just files or database and files?
Ideally, I should be able to login to your php user, delete everything I have access to and restoring it should be a inconvenience.

In summary, limit connection speed for some types of files, use a more effective firewall with rate-limiting and have a robust tested backup system.


Apologies for the long post and bandwidth usage :s - system admin / security are hobbies of mine. I sometimes help people who haven't a idea with system admin :) - It's also somewhat off topic so I'll leave it be with this post.
Good luck!

@paperdigits

This comment has been minimized.

Show comment
Hide comment
@paperdigits

paperdigits Dec 24, 2016

Is mediawiki a requirement? Or could it be a static site that gets auto built and published?

paperdigits commented Dec 24, 2016

Is mediawiki a requirement? Or could it be a static site that gets auto built and published?

@patdavid

This comment has been minimized.

Show comment
Hide comment
@patdavid

patdavid Dec 24, 2016

@paperdigits - the main site is static, but I think rawpedia is intended to be a wiki.

patdavid commented Dec 24, 2016

@paperdigits - the main site is static, but I think rawpedia is intended to be a wiki.

@paperdigits

This comment has been minimized.

Show comment
Hide comment
@paperdigits

paperdigits Dec 24, 2016

@kulga is your advice intended for pixls.us or for rawtherapee?

paperdigits commented Dec 24, 2016

@kulga is your advice intended for pixls.us or for rawtherapee?

@Beep6581

This comment has been minimized.

Show comment
Hide comment
@Beep6581

Beep6581 Dec 25, 2016

Owner

@kulga thank you for the advice. One of the reasons we're switching hosting companies is to get more access to the system which would let us do basic things like that.

@paperdigits MediaWiki is a requirement.

Owner

Beep6581 commented Dec 25, 2016

@kulga thank you for the advice. One of the reasons we're switching hosting companies is to get more access to the system which would let us do basic things like that.

@paperdigits MediaWiki is a requirement.

@Beep6581 Beep6581 assigned Beep6581 and unassigned Beep6581 Dec 29, 2016

@Beep6581

This comment has been minimized.

Show comment
Hide comment

@Beep6581 Beep6581 closed this Dec 29, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment