New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CurseMeta v2 updates/todo list #9

Open
dries007 opened this Issue Mar 5, 2018 · 2 comments

Comments

Projects
None yet
2 participants
@dries007
Owner

dries007 commented Mar 5, 2018

You can watch this issue if you want to know if/when sections are finished.

V2 is now live, under beta label. A live-edit staging environment can also be found on staging_cursemeta.dries007.net.

With the v2 python rewrite, all the previous aggregation/stats/history functionality was lost.
Bringing all of it back will take time, but will be done over time.

Feel free to comment and note what you need. New functionality may also be requested here.

ToDo snapshot as of March 14th

  • Referral link: https://workflowy.com/invite/4dfc7739.lnx (Gives me more free space to add stuff!)
  • Keep historic DL counts
    • PopularityScores?
    • The sooner the better, less data loss
    • I have the historic data still (Write import script)
    • API
      • Diff downloads (from-to timestamp)
      • Get total downloads per game/type
    • Keep up to date with feeds only?
    • Fixed interval data points or detect a change on every update?
      • I think more fixed interval, probably less data. Maybe even only do changed data?
  • Status page
    • Internal statistics such as
      • Threads running (?)
      • Uptime global/thread (?)
      • Requests served global/thread (?)
      • Task queue size
      • Server load
      • Approx bandwidth used
    • Any other stats?
    • Should this be public?
  • Web UI
  • DB query API
  • Uniform output API (Post process everything)
    • Fix typos (GameVersion is missing the R in some places)
    • Add Addon data to File objects (so less requests are needed)
    • Selective return data
      • Jenkins API like ?
    • Re-processed (or similar but DIY) raw feeds, preferably compressed.
  • Statistics pages (Graphs & tables!)
    • Old version: https://cursemeta2.dries007.net/stats/graphs.html
    • Maybe take some inspiration from MCStats.org ?
    • Stuff to stat (Global & per game, type, author(primary & all), gameversion)
      • nr of games
      • nr of addons
      • nr of files
      • nr of authors
      • Downloads
      • Filesizes
      • PopularityScores
      • GameVersions
      • Newest files
      • Newest addons
      • Newest authors (?)
      • Highest downloads 24h/7d/1m/
  • Separate out UI and data
    • Now the analysis of the response gets triggered by the CurseClient module, but is processed in the Frontend module (as part of tasks)
    • Make a more modularized system (would also support better tests)
  • File downloads (on the server)
  • Archive
    • We are not in the business of archiving, but supporting others is OK.
    • Write a script people can use to keep an archive (preferably with options).
  • Crazy idea: Draw a huge (interactive?) graph of the authors - addons relations + file dependencies?
  • Write the history of CurseMeta somewhere, because it's interesting.
  • Write tests (meh)
    • Get a list of troubling addon/file ids to test on (weird characters, no files, etc)
  • Pick a theme song.
@fiws

This comment has been minimized.

fiws commented Mar 6, 2018

My wishlist:

  • Cached API (quick win: set Cache-Control and let Cloudflare do it) – The old api was soooooo much faster
  • Addon -> LatestFiles -> Dependencies should always return an array, never null
  • Provide full mod list as tar.xz again. I wanted to use them for a local db.. :(
@dries007

This comment has been minimized.

Owner

dries007 commented Mar 10, 2018

  • Caching is now done with Redis locally (1h) + Cloudflare (4h).
  • Login tokens are now shared & cached (Redis) & is kept live at all times.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment