Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Convert to Go-lang based server #1

Merged
merged 32 commits into from Sep 7, 2015
Merged

Convert to Go-lang based server #1

merged 32 commits into from Sep 7, 2015

Conversation

bign8
Copy link
Owner

@bign8 bign8 commented Jul 29, 2015

Problem App Engine takes too long to spin up Python instances
Solution Switch to Go-lang
Status Complete

bign8 added 23 commits July 28, 2015 20:50
@bign8
First stroke at porting to golang
ce446e6
@bign8
Adding readme finally
6eda1ee
@bign8
Toying with entities and json in golang
ab6337c
@bign8
Reading entities already in datastore
6cc006d
@bign8
Adding parallel fetching of resources + monitoring
f1a8d2b
@bign8
Optimizing datastore reads to not use a ton of "Small Operations"
a012259
@bign8
Optimizing key storage and adding memcache
fbd907c
@bign8
Abstracting entity keys to a go module
80d581c
@bign8
Not pulling imgs already attached to objects
d2c83f1
@bign8
Single return point for random
0ab0549
@bign8
Adding xml parsing to cron objects
fdc9dcf
@bign8
Using memcache Codecs
5cec194
@bign8
Single Feed Processing Function
be1b768
@bign8
Storing posts in datastore with cron
1b62045
@bign8
Cron processing todo on defereds
92437e7
@bign8
Removing dependency on Google font CDN
7b4cb80
@bign8
Fixing reload_list problem
84de9e0
@bign8
Cleaning posts as they are stored
0684d71
@bign8
Moving error author img into repo
489b3d1
@bign8
Making cron work on appengine
a9e0b7b 
Aparently pulling the entire datastore across the wire is slow.
- Moving GUID numbers into the Key numeric ID
- Storing all new posts in PostNew
- Deployed code still has debug on... oops
@bign8
Batching Post puts
c7ead36
@bign8
Improving keycache implementation
b148e1d
@bign8
Removing fake loader
34b821a
@bign8 bign8 self-assigned this Sep 4, 2015
@bign8
Copy link
Owner Author

bign8 commented Sep 7, 2015

Verification

Currently there are no automated tests covering this repository, so manual testing is necessary

  • Deployed to AppEngine
  • Verified cron is running w/o errors
  • Verified api is returning expected values
  • Verified items show up in datastore

Items to note

  • Cron is still running in debug: this means only the first page of posts will be parsed every 6 hours
  • This is a total re-write of the server core (Python -> Golang)
  • Datastore still contains a large amount of Post types from the python cron (should delete)
  • Many improvements to come including, but not limited to:
    • Caching Post entities when pulled from datastore
    • More advanced post cleaning / filtering
    • UI improvements
    • Data Mining and custom user experiences

Sign-Offs

Dev +1: @bign8
QA +1: @bign8

bign8 added a commit that referenced this pull request Sep 7, 2015
@bign8
Merge pull request #1 from bign8/golang
60944d7 
Convert to Go-lang based server
@bign8 bign8 merged commit 60944d7 into master Sep 7, 2015
@bign8 bign8 deleted the golang branch September 13, 2015 15:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@bign8 bign8

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@bign8