Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make a staticman app instead of staticmanapp to avoid reaching quotas #243

Closed
robinmetral opened this issue Dec 11, 2018 · 64 comments

Comments

Projects
None yet
@robinmetral
Copy link

commented Dec 11, 2018

The public instance of Staticman is in trouble: API calls are now regularly hitting the Github quotas of 5000 hits per hour per user.

The issue is that every comment on any site using this instance is going through the staticmanapp user, therefore staticmanapp easily reaches its API quotas.

This causes the problems that have been repeatedly mentioned in #227, #222, or #242 for example.

I got in touch with the Github staff about this and here's their recommendation:

instead of using a single account to make all those requests, you should build an app so that your users can authorize/install the app. That way, the rate limits will scale much better -- for OAuth Apps, each user has their own quota and for GitHub Apps each installation has its own quota. So, your total rate limit would scale with the number of users instead of being static, which is what you want -- you want the limits to grow with your userbase.

@eduardoboucas you know your software - would this be possible?

@robinmetral robinmetral changed the title Make a staticman app instead of hitting the Github API to avoid hitting quotas Make a staticman app instead of staticmanapp to avoid reaching quotas Dec 11, 2018

@erised

This comment has been minimized.

Copy link

commented Dec 12, 2018

@robinmetral
I am in process of creating an GitHub app that tries to work as staticman. It is still in development and is in a private repository. I can extend invitation if you want to look or contribute.

I expect it to be completed by end of this month.

@robinmetral

This comment has been minimized.

Copy link
Author

commented Dec 12, 2018

Good to hear @erised!
Let me know when it's ready, I can help with testing 🙂

@erised

This comment has been minimized.

Copy link

commented Dec 12, 2018

@robinmetral Sure. I will let you know.

@casually-creative

This comment has been minimized.

Copy link

commented Dec 14, 2018

Just a friendly wake-up call to @eduardoboucas. Staticman is awesome, but right now, it's hitting its limits and people are developing alternatives. Please find the time to develop a solution so we can continue using this fantastic app without this very annoying limitation.

@eduardoboucas

This comment has been minimized.

Copy link
Owner

commented Dec 14, 2018

Hi everyone. I'm sorry that some people are frustrated with the project, I can relate to that. But please remember that the code is fully open-source, which means anyone can simply run their own instance, with their own GitHub account, and bypass all these limitations. The issue we're seeing here is really a problem with the free, public instance I decided to host for everyone. In hindsight, this probably wasn't the best of ideas, because it puts pressure on me to be the sole gatekeeper of this service.

I've not abandoned the project and I'm thinking of solutions to solve both problems: the problem that a single GitHub account acting on behalf of everyone isn't maintainable, and the problem that in the current scheme of things, where the public instance that I run is the centrepiece of the project, I represent a bottleneck.

For example, I'm keen on the idea of rebuilding Staticman as a Netlify function, so that effectively everyone is running their own instance (for free) rather than relying on a centralised service (which I'm covering the costs for). In this scenario, people would provide access to their own GitHub account, which commits would be made from, thus removing the issue of quota limits.

All I can say is that your patience is much appreciated and I'd love to hear everyone's thoughts on how we can make this more manageable for everyone.

@erised

This comment has been minimized.

Copy link

commented Dec 14, 2018

@eduardoboucas The best solution in my opinion, would simply be a GitHub app.
GitHub Apps also have their call limits, but they have this per installation.

If repo A & repo B, install an app on their repos, each gets 5k hits per hour.
I have already started working on this, I will present it here when it is prsentable.

Till then, I am also open to any discussion that how we can make it work for everybody.

@robinmetral

This comment has been minimized.

Copy link
Author

commented Dec 15, 2018

Thanks for the updates @eduardoboucas ! 🙌

A Netlify function would be great as a simpler solution to people wanting to self-host their instance.

However one of the things I love with Staticman is how simple it is to set it up only using a Github repo!

I think that many people would benefit from such a "centralized" service, be it for testing or to allow comments on small websites and blogs (GH pages for example), where setting up Netlify+AWS Lamba to run a self-hosted instance seems like a lot of trouble.

In this case a Github app would probably be the best solution! @erised can you consider making your working repo public so that we can take a look and maybe contribute? 🙂

@erised

This comment has been minimized.

Copy link

commented Dec 15, 2018

@robinmetral Please give me a day or two.

I want to finish a few tasks by myself.
I don't want myself to be an embarrasment as this is my first project.
I just need it to be presentable with a few features. Then, i will work on it with everyone.

@eduardoboucas

This comment has been minimized.

Copy link
Owner

commented Dec 15, 2018

I don't want myself to be an embarrasment as this is my first project.

No reason to feel embarrassed at all! We all appreciate the effort you're putting in. Whenever you feel comfortable showing your code, I'm happy to review and help you change anything that needs tweaking.

@erised

This comment has been minimized.

Copy link

commented Dec 15, 2018

@eduardoboucas Thank you. It means a great deal. I am looking over staticman's code all the time to see how everything works.
I am sure will let you know.

@maciek134

This comment has been minimized.

Copy link
Contributor

commented Dec 16, 2018

@erised if you need any help I'd be happy to assist as well.

@casually-creative

This comment has been minimized.

Copy link

commented Dec 17, 2018

Thanks for your reply @eduardoboucas. Staticman running in a netlify function sounds like a very good idea. I'm looking forward to you testing this out and, if found feasible, giving us feedback on how to set it up for ourselves. Keep up the good work :D

@rliebling

This comment has been minimized.

Copy link

commented Dec 29, 2018

If i understand things correctly, Netlify provides a way to translate form submissions into the Functions (aws lambda events). However, the free plan limits this to 100 form submissions per month (even though the Functions limit for the free plan is much higher). Just FYI.

Meanwhile, if i understand things correctly (and I may not) the /connect controller is invoking the GET /user/repository_invitations API method (https://developer.github.com/v3/repos/invitations/#list-a-users-repository-invitations) which will only return the first 30 invitations. I do not see any handling of pagination (although possibly it's built into the github client library you are using by default, or i am missing some setup someplace). Thus, many calls to connect are likely to fail with Invitation Not Found if there are more than 30 queued up. And, their continuing retry attempts help exhaust the API limits.

If the above analysis is correct a few things would greatly help:

  1. set the page size to 100 (the max github allows)
  2. handle pagination. if the api returns them in FIFO order, then perhaps starting from the tail would help - which really means get the first page, then jump to the last page (the link should be provided in the api response) and work toward the front until found. This would favor recent invitations.
  3. Possibly, when traversing the list of invitations, reject all those older than N hours.
  4. Cache the list of invitations for N minutes to reduce API hits to github. Suspect that if you reject the old invitations then the current list at any given time should be maintainable at <100 (so a single page)

Sorry I don't have time to do it myself - but maybe someone else can contribute improvements here. My suggestion would be to start with just setting the per_page parameter and rejecting stale invitations (to unclog things and keep them unclogged). Then pagination support and caching probably become unnecessary,. Although users with pending invitations who have not yet given up will have to re-invite, at least they will likely be successful at that time.

Update: Was late when i originally posted this. While lying in bed i wondered why instead you don't just accept all invitations. Each time you getRepoInvites, just accept them all. This will keep the queue low. And, if you just schedule this to happen say each minute, then you won't need folks to hit the endpoint to connect at all. I'm assuming the point of this was only to trigger the invite acceptance.

@maciek134

This comment has been minimized.

Copy link
Contributor

commented Dec 30, 2018

@rliebling great analysis, I didn't even think about that. I'm more than happy to provide a PR for this.

@eduardoboucas

This comment has been minimized.

Copy link
Owner

commented Jan 12, 2019

Hi all.

I had a go at implementing Staticman as a GitHub App, which should fix many of the issues people are seeing at the moment. Can I ask for some volunteers to help me test it? Here's how:

  1. Remove staticmanapp as a collaborator
  2. Go to https://github.com/apps/staticman-net and install the application on your repository
  3. Submit a comment to the new v3 endpoint, using dev.staticman.net as the base URL – i.e. https://dev.staticman.net/v3/entry/github/[USERNAME]/[REPOSITORY]/[BRANCH]

Any help is much appreciated.

@rliebling

This comment has been minimized.

Copy link

commented Jan 12, 2019

Hi @eduardoboucas

Great to hear the news from you!

Not sure if i've just done something wrong but i'm getting 500 errors. I removed staticmanapp as a collaborator. I installed the staticman-net github app with access to my repo. And, i tried submitting a comment to https://dev.staticman.net/v2/entry/rliebling/my_blog/master/comments. After retrying a couple times i tried curl'ing the /v2/connect/rliebling/my_blog endpoint and also got a 500 response.

Note that i'm using Hugo with the engimo them which has staticman support built in, but i've never successfully used staticman before (as my invitation was "not found"). Also I'm testing using Hugo on my localhost - assume the post-id stuff wouldn't cause the problem, but i mention in case i'm wrong. The request that's failing is (as copied from chrome debugger, removing user agent and cookies:

curl -i  'https://dev.staticman.net/v2/entry/rliebling/my_blog/master/comments' -H 'authority: dev.staticman.net' -H 'cache-control: max-age=0' -H 'origin: http://localhost:1313' -H 'upgrade-insecure-requests: 1' -H 'content-type: application/x-www-form-urlencoded' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-US,en;q=0.9' --data 'options%5BpostId%5D=dc33308b4aef09b40e86a6783d501abd&options%5Bredirect%5D=http%3A%2F%2Flocalhost%3A1313%2Fposts%2Fmore_on_tech_debt%2F%23submission-success&options%5BredirectError%5D=http%3A%2F%2Flocalhost%3A1313%2Fposts%2Fmore_on_tech_debt%2F%23submission-failure&fields%5Bhoneypot%5D=&fields%5Bpermalink%5D=%2Fposts%2Fmore_on_tech_debt%2F&fields%5Bparent_id%5D=&fields%5Bcontent%5D=test+comment&fields%5Bauthor%5D=rich&fields%5Bemail%5D=rliebling%40gmail.com&fields%5Bsite%5D=https%3A%2F%2Fexample.com' --compressed
@eduardoboucas

This comment has been minimized.

Copy link
Owner

commented Jan 12, 2019

Thanks for the feedback. I’ll try to debug it tonight using your sample request and will come back with my findings.

Thank you all for your patience.

@eduardoboucas

This comment has been minimized.

Copy link
Owner

commented Jan 12, 2019

@rliebling Your site doesn't seem to be configured properly. I don't see a configuration file for Staticman on https://github.com/rliebling/my_blog.

@rliebling

This comment has been minimized.

Copy link

commented Jan 13, 2019

Ah! My bad! Had only configured locally and never pushed to GH as my invite had not been accepted. And, did this testing having forgotten all about that!

Sorry - I should have checked and figured this out myself.

I've fixed that now, however, and still getting 500 response. I don't want you having to go about debugging my config if you think that's likely the issue. I'll try looking at the code to understand better what it's doing. But, one quick thing to confirm: if i configure path: "data/comments/{options.postId}" inside staticman.yml should i need that directory/path to already exist?

Note: i've also enabled commenting on my live site now -- just not working yet (eg https://rich.liebling.us/posts/more_on_tech_debt/)

@eduardoboucas

This comment has been minimized.

Copy link
Owner

commented Jan 13, 2019

@rliebling Your config is fine, it was an issue with an environment variable on the development instance. I've fixed it, tested again and it seems to be working.

You can see a submission here: rliebling/my_blog@e36206d

@rliebling

This comment has been minimized.

Copy link

commented Jan 13, 2019

@eduardoboucas Thanks so much for this project, moving it to a github app, and for your help here!

@simonarnell

This comment has been minimized.

Copy link

commented Jan 13, 2019

Hi all.

I had a go at implementing Staticman as a GitHub App, which should fix many of the issues people are seeing at the moment. Can I ask for some volunteers to help me test it? Here's how:

  1. Remove staticmanapp as a collaborator
  2. Go to https://github.com/apps/staticman-net and install the application on your repository
  3. Submit a comment as usual, but use https://dev.staticman.net instead of https://api.staticman.net as the base URL.

Any help is much appreciated.

Thanks @eduardoboucas. This seems to be working great for me on my project. https://github.com/simonarnell/GDPRDPIAT

@eduardoboucas

This comment has been minimized.

Copy link
Owner

commented Jan 13, 2019

I've updated the comment above to point to the new v3 endpoint. The idea is that people will carry on using v1 or v2 endpoints if they're using the legacy staticmanapp authentication method, whilst people that have installed the new GitHub App will use the v3 endpoints.

@snirp

This comment has been minimized.

Copy link

commented Jan 25, 2019

I can confirm that V3 works well for me.

@davidomarf

This comment has been minimized.

Copy link

commented Jan 30, 2019

I just got it working on my website. Thanks all of you for being so awesome and wholesome!

@willymcallister

This comment has been minimized.

Copy link
Contributor

commented Feb 4, 2019

Staticman App requires moderated comments. My site (https://spinningnumbers.org/a/staticman.html) was not automatically rebuilding with unmoderated comments. Here is the full diagnosis from GitHub Support...

It looks like your site builds are failing with the following error:
Validation failed: User must be a human
To avoid abuse, GitHub Pages sites can't be automatically rebuilt when a GitHub App pushes to a repository. As StaticMan is a GitHub App it isn't able to automatically trigger a rebuild of your site.

I notice that this application has a moderation-mode, that will create pull requests for you to merge manually. Enabling this mode will let this app create pull requests, and will then trigger a build when you manually merge them.

I'd recommend reaching out to the project maintainer for more help on this, as this will need to be something that is changed on their side.

mmistakes added a commit to mmistakes/made-mistakes-jekyll that referenced this issue Feb 7, 2019

@zinefer

This comment has been minimized.

Copy link

commented Feb 12, 2019

I use a botpot field that is intended for a user to leave blank. However, these blank fields are appearing in the comment files added by Staticman. Is it possible to strip empty fields from the comment files?

Maybe a feature request to strip blank fields that are not in allowedFields would be more appropriate.

@robinmetral

This comment has been minimized.

Copy link
Author

commented Feb 12, 2019

@zinefer @willymcallister maybe you should post new issues for this!

Your problem will have much more visibility than as a message on a thread with 50+ comments, and it'll let us keep issues organized 🙂

Closing this for now, feel free to reopen if relevant 👋

freefallcid added a commit to freefallcid/dominicreich-jekyll-2019 that referenced this issue Feb 16, 2019

@lazywinadmin lazywinadmin referenced this issue Feb 20, 2019

Closed

New comment system #8

1 of 3 tasks complete
@domguard

This comment has been minimized.

Copy link

commented Feb 25, 2019

Testing the v3 api I only get this error:
curl -d "fields[name]=dom&fields[email]=truc@truc.net&fields[message]=test" -X POST https://dev.staticman.net/v3/entry/github/domguard/onlefay-comm/master/comments
{"success":false}
Does my github repository need to be public for the app to be able to push comments ?

@MyGuySi

This comment has been minimized.

Copy link

commented Mar 10, 2019

I started to add Staticman to my project today and ran into troubles with V2, presumably because of the rate limiting. Decided to give V3 a go and it just worked great first time 👍

Might be worth considering updating the docs on the website because V2 was pretty much unusable for me.

@robinmetral

This comment has been minimized.

Copy link
Author

commented Mar 10, 2019

Agreed @MyGuySi, I'm sure @eduardoboucas would be open to a PR 🙂

@valzi

This comment has been minimized.

Copy link

commented Mar 16, 2019

Hi all.

I had a go at implementing Staticman as a GitHub App, which should fix many of the issues people are seeing at the moment. Can I ask for some volunteers to help me test it? Here's how:

  1. Remove staticmanapp as a collaborator
  2. Go to https://github.com/apps/staticman-net and install the application on your repository
  3. Submit a comment to the new v3 endpoint, using dev.staticman.net as the base URL – i.e. https://dev.staticman.net/v3/entry/github/[USERNAME]/[REPOSITORY]/[BRANCH]

Any help is much appreciated.

I followed the instructions except that I do not understand step 3. I typed in the URL and got "Cannot GET /v3/entry/github/valzi/light-transmuter/master". Where am I supposed to go to submit the comment?

@robinmetral

This comment has been minimized.

Copy link
Author

commented Mar 17, 2019

@valzi this URL is the one that your comment form has to POST to. The Get started doc is a good place to start building with Staticman 🙂

@valzi

This comment has been minimized.

Copy link

commented Mar 17, 2019

Thanks, I get it now.

bazbt3 added a commit to bazbt3/bazbt3.github.io that referenced this issue Mar 20, 2019

bazbt3 added a commit to bazbt3/bazbt3.github.io that referenced this issue Mar 20, 2019

Delete readme.md
Removed Staticman integration:  eduardoboucas/staticman#243
@pacollins

This comment has been minimized.

Copy link

commented May 11, 2019

I don't have reCAPTCHA set but keep getting this error:

{"success":false,"message":"Missing reCAPTCHA API credentials","rawError":{"_smErrorCode":"RECAPTCHA_MISSING_CREDENTIALS"},"errorCode":"RECAPTCHA_MISSING_CREDENTIALS"}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.