HTTP 503 response from urlreq.appspot.com

[ghjansen]

The server responds with HTTP 503 since yesterday, for the request https://urlreq.appspot.com/req?method=GET&url={url}

[ghjansen]

@izuzak Please, can you elaborate on the steps to have urlreq running on a new instance at Google Cloud?

I would like to create an instance for me for better handling quota, but since Python versions are distinct between environment's default and your project, may some hacking is needed.

[Cipher-Coder]

If you clone the repo and make a few changes, you can get it to work. I set it all up this morning using Google App Engine. I changed a few lines in the app.yaml file.
When I tried to deploy it, there was a couple errors. I guess App Engine does not use the same info now. But I just took out the first two lines. Then I changed the urlreq.py file. I changed the reference to his repo and just put a reference to mine.
I use Linux and it already has Python installed. But I did download the Google App Engine SDK and python tools. This way I was able to interact with Google App Engine through the command line. It really was not hard. I got my instance up and running in about an hour. If you need a reference => take a look at my repo. Even clone it if you have to, then the only thing you would have to change is the line in the python file pointing to the repo then deploy it to your Google App Engine account. @Cipher-Coder and the repo is: mattazurl.
Hope this helps.

[ghjansen]

Thank you @Cipher-Coder
I just checked your repo and the changes are indeed minimal. I did something similar myself, but I am new to Python and Google Cloud environment. Can you help me with the following questions:

1. What is the Python version of the virtualenv you created for your app?
2. Did you installed Google App Engine SDK by downloading it via console and using install.sh?
3. The steps that you took diverge too much from the Interactive tutorial called "Try App Engine" using Python (access on upper right corner from inside dashboard) or maybe this one ?

I am already using the command gcloud but the commands gcloud components install app-engine-python and gcloud components install app-engine-python-extras outputs only All components are up to date.

Even though, using Python 3 triggers Exception from within webapp2.

Using Python 2, I am facing all kinds of module errors, so I had to manually pip install modules like webapp2 and webob, which cascaded into different errors. I am most certainly missing something.

[Cipher-Coder]

I will be honest... I am not a Python expert. In fact, I am a Front End Developer. I use this in a dependency I have in a Chrome and FF extension that I authored and when it went down with the 503, it broke a portion of my app. So I had to do something to get back up and running quick. I know a little python from years ago when I first started. I tell you this because I am not sure I will have all the answers you are looking for. I will walk through what I did so that hopefully you can reproduce.

1. I went to App Engine Downloads and downloaded just the SDK for python 2.7. I followed the Debian instructions. I did do the python extras and stuff but am pretty sure I did not need them. There was no install.sh script. Just a bunch of curl and apt-get commands. I did read you already have them so that should be good.
2. After download I went to the Google Cloud interface and since I already have a bunch of projects there I just created a new one called mattazurl so I was not using a client project.
3. I went back to the command line and did the gcloud init command and authorized the cloud platform on my machine and then it asked me to select a project, which I selected the one I had just created.
4. I then went back to the repo I downloaded - and to be honest, I just opened the files with VS Code. In fact VS Code recommended I download an extension to work with Python, which I did, and I made the edits to the files. I say this because if there were errors being shown from a full fledged IDE setup, I never saw them since I was just using a text editor.
5. After I made the edits, I did look at one of those how-to projects but only to see some of the commands.
6. I then cd /mattazurl/src and opened a terminal. This is where I actually used the deploy command which I copied from the how-to project. I used gcloud app deploy app.yaml --project mattazurl which is why I made sure I was in the src folder so I could use the app.yaml file.
7. Once I did this it started throwing errors in the cli. It just told me that it did not need the application: urlreq-hrd so I deleted that. Tried the deploy command again and it threw the error that it did not need the version. So I deleted the version: 1 line and tried again.
   It worked.
   The environment is python 2.7
   In my instance on App Engine it tells me my runtime is python27 and environment is standard.
   If you are looking for any insight into the code, I am not all that familiar. In my instance it tells me there are no application errors and you can see what I actually changed.
   I hope this helps.

[ghjansen]

@Cipher-Coder thank you so much! By following your steps I managed to get it working. I can't tell exactly what did I miss before tho.

As a first contact with gcloud I would be lying to say it was intuitive, but matters not, it is free and it works. Soon I intend to explore the platform without the rush of having dependencies breaking apart, like this week, so I will probably get more familiar with all.

I saved your steps and added some notes of mine for a next time. Thank you so so much for the help! You really helped a lot! :)

By the way, if you have awareness of a opportunity for a remote backend dev (Java), I would be glad to know. :)

Thanks once again! Take care.
I'm closing this issue now.

[izuzak]

Sorry for not responding to this sooner, @ghjansen! And thank you, @Cipher-Coder for helping out here.

I have to say that I haven't touched the deployment of this project in years. It's been happily running on AppEngine for years without me touching it. It's possible that some of the AppEngine features have been deprecated or changed so you might not be able to deploy it as is.

@Cipher-Coder thanks for the detailed instructions -- I'll probably be using those myself if I decide to make any changes in the future.

But I do have one question for both of you @Cipher-Coder @ghjansen -- how exactly are you using this project? The fact that the quota has been drained means that the project is being used, so I'm wondering how/why folks are using it. It was mostly a personal project I worked on for learning, not something for real-world production use, so I didn't expect folks would be using it as a dependency. If the tool is useful, I'm wondering if it would be helpful to improve it and run it somewhere with a higher quota. 💭

[ghjansen]

Thanks @izuzak

In my case, I was looking for a portfolio (other than GitHub's profile page), so I found https://github.dev and decided to try. Apart from the supported features, I missed the contributions calendar, but I read at github/personal-website#32 about github-calendar, which is currently using your https://urlreq.appspot.com.

In the beginning, all was working well for me, but I tested in the day after of having my portfolio online and the contributions calendar was gone. The HTTP 503 was the root cause.

As you may notice in github/personal-website#32, I was not the only one looking for the contributions calendar, and if you actually see the numbers of this repo (used by, stars, forks) there is a reasonable amount of people involved with https://urlreq.appspot.com, just from that source.

Anyway, I see that Bloggify/github-calendar#54 where you first suggested spinning up a new instance was back in 2018. I don't know your quota drain peaks, but https://github.dev was released at the beginning of this year. One possibility would be that the interest in https://github.dev let to github-calendar and consequently drained the quota, just like it happened to me. But I'm being totally speculative here.

In any case, a higher quota would be appreciated, if that turns out to be possible/viable for you. I also raised the concern for a better solution at Bloggify/github-calendar#54.

Thank you to consider improving, and congrats on your project! :)

[Cipher-Coder]

Thank you and I am sorry @izuzak

I use the github-calendar library in my Dev Tabs Chrome Extension and in my Dev Tabs Firefox Extension. I had planned on re-writing the lib for a while now but never seem to get around to it. Either way.. I did not realize this library that is out there in the wild, and pretty easy to find, completely relied on someone else's free App Engine plan. In fact... If I had known that, I would have re-written the lib earlier or just rolled my own when I created the extension.
Problem solved now. I am using my own instance in both the extensions now and I apologize for running your plan out, as I am afraid I am part of the problem

I am glad you were able to get everything going and I was able to help @ghjansen

[izuzak]

Thanks so much for that context, @ghjansen @Cipher-Coder -- that's very helpful. ⚡️ So, yeah, it seems that the github-calendar library might be the main consumer of the quota. I'll give this some thought and maybe talk to @IonicaBizau to see if we can come up with a better solution.

@Cipher-Coder no need to apologize, you did nothing wrong. The https://urlreq.appspot.com project was built to be used, and hitting quota limits is a good problem to have -- it means that there's a non-insignificant demand for it.

[admhpr]

Hi @izuzak, I ended up here after noticing some over quota issues. I am the author of https://www.npmjs.com/package/vue-github which is a personal project that has typically around 50 downloads a week.

There isn't a ton of demand for the package but it does use https://urlreq.appspot.com under the hood, so it has also contributed to the over quota issue.

I'd be happy to help to with coming up with a better solution and share some of the work/costs.