Add API Update Scripts #40

South-Paw · 2018-09-01T07:54:42Z

@Mulchman, @will-janz and @andersbrandt if anyone has a chance to look over these changes and provide feedback that'd be awesome.

Changes in v5.0.0

Update script that uses mobile endpoints now exists, update the repo using yarn update
New items will need to have their name added to the update/data/categories.js file so that they receive a weapon category (otherwise they'll get an 'Unknown' category)
Package now includes both the .js files and the formatted .json files when installed.
Removed acquisition key from items - it's going to be too much work to maintain this going forward so I'd opting to remove it.

Remaining Work

Fetch category information from wikia module
Images for items
Fix travis ci build there's some issue with node-lua package.
Add unit tests & raise code coverage where possible

Fixes #40, fixes #39, fixes #38, fixes #37 and fixes #35

Note that the weapon items do not contain any category yet. Need to resolve all the undefines in the `supplementary.js` first.

South-Paw · 2018-09-01T07:55:34Z

I also have a question around whether I should implement fetching and resizing of the weapon images - however this will of course add heaps of megabytes to the file size of the package... thoughts, suggestions or ideas on this would be much appreciated.

andersbrandt · 2018-09-01T18:45:27Z

Nice touch with the missing items system, but the Venari Kavat does not contribute towards mastery rank. Keep the system though, might be needed in the future.

Mulchman · 2018-09-01T18:54:56Z

As far as I'm aware (and some others on Reddit, Venari does give Mastery): https://www.reddit.com/r/Warframe/comments/8dorwn/does_venari_have_mastery_associated_with_her/

…

On Sat, Sep 1, 2018, 11:45 andersbrandt ***@***.***> wrote: Nice touch with the missing items system, but the Venari Kavat does not count towards mastery rank. Keep the system though, might be needed in the future. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#40 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALNpqZiR_ZE-EoDGvrtxLUlWVS18mRp6ks5uWtXHgaJpZM4WWH90> .

South-Paw · 2018-09-01T23:18:22Z

Yeah, I was working from the wiki which says it does but isn't listed on player profiles: http://warframe.wikia.com/wiki/Kavat#Venari

South-Paw · 2018-09-01T23:57:26Z

@andersbrandt Regarding the acquisition data - is there any known endpoints or pages that provide a list of that information? I don't want to update things manually as it requires time and results in update delays

andersbrandt · 2018-09-02T06:23:02Z

I stand corrected :)

Regarding acquisition data - I changed my mind, that data are important, but this repo will be better off without manual steps for updates. I will add the data for acquisition in my own end instead.

Regarding images: I assume we are talking about adding a URL to an image, and not the actual image-file. If that's the case it is an most excellent idea :)

South-Paw · 2018-09-02T11:34:32Z

@andersbrandt I was thinking of adding the actual images to the /data/img folder.

Providing an endpoint/url for them is easy but not all the images that the mobile endpoints provide are sized correctly. They have to be run through a resizer for them to appear correctly so I was thinking of doing that and providing the resized images.

I'll look into exporting them in alternative means (such as data urls rather than pngs) and see what the bundle size comes out like?

Mulchman · 2018-09-03T17:46:08Z

I don't think the images themselves should be a part of this. A URL to the image, sure, but the actual data, no.

If you've ever messed with Destiny 1/2 data (and their APIs) they just provide the URL to weapons/armor/gear/etc.

hylte-brandt · 2018-09-03T19:33:20Z

src/update/updater.js

+        const unoderedInnerKeys = unordered[key];
+        const orderedInnerKeys = {};
+
+        Object.keys(unoderedInnerKeys).sort().forEach((innerKey) => {


typo "unodered"

Ah cheers, it was like 1am when I was reviewing this ;)

South-Paw · 2018-09-08T07:33:21Z

I've been messing around with the best way to do images for the repo the past week and I've got a few options available...

No images/urls

Without any image information being added to the data/json/... files, the base file size is 105KB (136KB on disk).

URLs only

Its super easy and lightweight to do this as I'm basically only adding a URL to each item object. The file size of the data/json files becomes 156KB (180KB on disk).

Implementing applications would need to fetch these items themselves from the endpoint as well as do their own resizing on the images as they are all 512x512 images but need to be resized to 512x342 to actually display correctly.

The item objects would look like the following;

{
    "id": "/Lotus/Weapons/Corpus/LongGuns/CrpShotgun/CrpShotgun",
    "name": "Arca Plasmor",
    "type": "Primary",
    "category": "Shotgun",
    "masteryRank": 10,
    "imageUrl": "http://content.warframe.com/MobileExport/Lotus/Interface/Icons/Store/CrpShotgun.png",
    "wiki": "http://warframe.wikia.com/wiki/Arca_Plasmor"
},

Image Data URLs

This is a heavy update operation and results in data urls being available in each item object. The file size of the data/json files becomes 75.6MB (75.6MB on disk).

However every one of these data urls has an image that is already resized and prepared for use in an application. The downside is there's no file for them in the repo, only a data url and the files become really really really long so diffs might be an issue.

{
    "id": "/Lotus/Weapons/Tenno/Archwing/Melee/ArchSwordHook/ArchHookSwordWeapon",
    "name": "Agkuza",
    "type": "ArchwingMelee",
    "category": "Archwing Melee",
    "masteryRank": 0,
    "imageUrl": "http://content.warframe.com/MobileExport/Lotus/Interface/Icons/Store/ArchHookSword.png",
    "imageData": "data:image/png;base64,iVBORw0KGgoAAA....",
    "wiki": "http://warframe.wikia.com/wiki/Agkuza"
  }

Saving image files to `data/img` folder (after resizing)

Another heavy process and results in resized image files being available in a folder (and referenced by each image object). The file size of the data/... folders files becomes 61.5MB (62.5MB on disk).

And if I run the images through a imagemin after saving them, the file size becomes 16.2MB (17.2MB on disk) which in the end seems pretty reasonable.

South-Paw · 2018-09-08T07:39:32Z

	File Size	on Disk	Resized?
No Images	105KB	136KB	n/a
URLs Only	156KB	180KB	no
Data URLs	75.6MB	75.6MB	yes
Image Files	61.5MB	62.5MB	yes
Image Files (+ minification)	16.2MB	17.2MB	yes
Data URLs from minified images	20.0MB	20.1MB	yes

So @Mulchman and @andersbrandt given the above, what are your thoughts?

Bear in mind, if I only provide urls the images are not sized correctly so the url is as good as useless (in my opinion only though).

I'm very tempted to add the resized and minified images to the repo at this stage.

andersbrandt · 2018-09-08T11:43:42Z

Image Files (+ minification) sound about right.
There is the option of scraping Wikia for images, i do that in my repo.
That procedure is rather time-consuming though.

Have you tested to convert to image data after minification?

South-Paw · 2018-09-10T11:11:45Z

@andersbrandt just made some quick changes and tested data urls from the minified images; 20.0MB json files (20.1MB on disk) - have updated table too.

You're also talking ~71,000 or more characters per image which is insane.

I'm keen to stick with the files at this stage.

* Swapped usage of `node-lua` to an updated one to see if it alleviates the build issues.

South-Paw · 2018-09-11T06:34:42Z

@andersbrandt @Mulchman I could do with some help on the travis ci front if either of you have time or suggestions on how to get it running again.

My hunch is simply that the ci doesn't have lua installed or on the path correctly - however I don't know much about linux or how to solve this particular issue.

andersbrandt · 2018-09-11T08:37:22Z

@South-Paw Aight, i will get some help to look into that

andersbrandt · 2018-09-15T06:42:53Z

victorliu/S4#45

victorliu/S4#46

https://github.com/kikito/middleclass/blob/master/.travis.yml

https://github.com/moteus/lua-travis-example

This reverts commit bb9e36f.

South-Paw · 2018-09-17T10:58:30Z

Right well I'm at an absolute loss on this and nothing I've been able to pull up via google has helped.

I've raised a stack overflow question (https://stackoverflow.com/questions/52366146/node-js-and-travis-ci-cant-find-lua-h-when-installing-node-lua-package) and hopefuly someone who knows what the heck is wrong can come save this PR.

I don't even know how I got into this point, but yeah.

This reverts commit 03fb96f.

resir014 · 2018-09-19T16:04:40Z

CI fixed. #41

Edit: Sorry it took so long btw @South-Paw. Work and all that.

Attempt to fix CI

South-Paw · 2018-09-20T08:18:37Z

Shoulda, coulda, woulda squashed the commits when I merged that ... but not really worried 👍

Thanks so much for your help @resir014.

I'll go about raising test coverage now that CI is working again - glad to see that getting it building on CI ended up being so easy 😆

South-Paw · 2018-09-20T09:48:42Z

@Mulchman and @andersbrandt do you have any comments on this or are we all happy if I go ahead and merge this PR? 😄

hylte-brandt · 2018-09-20T11:20:10Z

Just go man, incredible work, this project is now so much better :)

hylte-brandt · 2018-09-20T11:22:34Z

My only objection is Venari, the companion to Khora.
The actual number of ranked items will not correspond to the number displayed in-game if Venari are still among the items. However, it's a small task to remove that entry for projects that use this repo as a base.

South-Paw added 6 commits August 31, 2018 18:35

WIP first cut of the updater script

e7d3b2b

Updater progress, now produces nice json files.

3e7deda

Note that the weapon items do not contain any category yet. Need to resolve all the undefines in the `supplementary.js` first.

Changes to updater to add manually updated weapon category lists

201a522

Update package javascript to use the .json files.

52463f9

Update version and readme.

fdc5066

Add comments to update scripts so they're a bit easier to follow.

4f47fed

South-Paw added the enhancement ✨ label Sep 1, 2018

Repository owner deleted a comment from coveralls Sep 1, 2018

South-Paw added 2 commits September 4, 2018 01:35

Use wiki data to populate weapon categories

2b25733

Bump travis ci version

3a43cc5

hylte-brandt reviewed Sep 3, 2018

View reviewed changes

Fix typo

e90a8d7

South-Paw added 2 commits September 10, 2018 23:40

Add pre-resized image for each mastery item.

22480f1

Fixed image location paths relative to the package.

acc0a46

* Swapped usage of `node-lua` to an updated one to see if it alleviates the build issues.

Update

7b3cf47

South-Paw added 7 commits September 17, 2018 21:24

Test adding Lua to CI

1cb324a

Add Lua to CI

bb9e36f

Revert "Add Lua to CI"

c924130

This reverts commit bb9e36f.

Lua CI test

5109256

Lua CI test

42c6453

Lua CI test

4203670

Reverting lua changes

7fa4302

resir014 added 10 commits September 19, 2018 20:44

Just Use Node-LuaJIT™

03fb96f

sudo build luajit for me please

9d1efc2

enable sudo -su before install

197ea5a

I don't even know how I got into this point, but yeah.

add luajit as package

64a3c11

Revert "Just Use Node-LuaJIT™"

f683588

This reverts commit 03fb96f.

add libluajit-5.1-dev

745bff7

only include libluajit-5.1-dev

ecdec78

switch back to standard node-lua

b9f7ab4

use the forked version of node-lua

48350b0

yes, I'm that dumb.

ca7955e

resir014 and others added 2 commits September 20, 2018 01:46

no need for sudo now (hopefully)

0eadc2f

Merge pull request #41 from resir014/api-generated

fb87b78

Attempt to fix CI

South-Paw added 3 commits September 20, 2018 21:36

Update readme and lockfile

6499aa0

Update warframe version

85a9c05

Add tests and fix some comments

78babbe

South-Paw merged commit 99551b2 into master Sep 21, 2018

South-Paw deleted the api-generated branch September 21, 2018 23:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add API Update Scripts #40

Add API Update Scripts #40

South-Paw commented Sep 1, 2018 •

edited

Loading

South-Paw commented Sep 1, 2018

andersbrandt commented Sep 1, 2018 •

edited

Loading

Mulchman commented Sep 1, 2018 via email

South-Paw commented Sep 1, 2018

South-Paw commented Sep 1, 2018

andersbrandt commented Sep 2, 2018

South-Paw commented Sep 2, 2018 •

edited

Loading

Mulchman commented Sep 3, 2018

hylte-brandt Sep 3, 2018

South-Paw Sep 4, 2018

South-Paw commented Sep 8, 2018 •

edited

Loading

South-Paw commented Sep 8, 2018 •

edited

Loading

andersbrandt commented Sep 8, 2018

South-Paw commented Sep 10, 2018 •

edited

Loading

South-Paw commented Sep 11, 2018

andersbrandt commented Sep 11, 2018

andersbrandt commented Sep 15, 2018 •

edited

Loading

South-Paw commented Sep 17, 2018 •

edited

Loading

resir014 commented Sep 19, 2018 •

edited

Loading

South-Paw commented Sep 20, 2018 •

edited

Loading

South-Paw commented Sep 20, 2018

hylte-brandt commented Sep 20, 2018

hylte-brandt commented Sep 20, 2018

Add API Update Scripts #40

Add API Update Scripts #40

Conversation

South-Paw commented Sep 1, 2018 • edited Loading

Changes in v5.0.0

Remaining Work

South-Paw commented Sep 1, 2018

andersbrandt commented Sep 1, 2018 • edited Loading

Mulchman commented Sep 1, 2018 via email

South-Paw commented Sep 1, 2018

South-Paw commented Sep 1, 2018

andersbrandt commented Sep 2, 2018

South-Paw commented Sep 2, 2018 • edited Loading

Mulchman commented Sep 3, 2018

hylte-brandt Sep 3, 2018

Choose a reason for hiding this comment

South-Paw Sep 4, 2018

Choose a reason for hiding this comment

South-Paw commented Sep 8, 2018 • edited Loading

No images/urls

URLs only

Image Data URLs

Saving image files to data/img folder (after resizing)

South-Paw commented Sep 8, 2018 • edited Loading

andersbrandt commented Sep 8, 2018

South-Paw commented Sep 10, 2018 • edited Loading

South-Paw commented Sep 11, 2018

andersbrandt commented Sep 11, 2018

andersbrandt commented Sep 15, 2018 • edited Loading

South-Paw commented Sep 17, 2018 • edited Loading

resir014 commented Sep 19, 2018 • edited Loading

South-Paw commented Sep 20, 2018 • edited Loading

South-Paw commented Sep 20, 2018

hylte-brandt commented Sep 20, 2018

hylte-brandt commented Sep 20, 2018

South-Paw commented Sep 1, 2018 •

edited

Loading

andersbrandt commented Sep 1, 2018 •

edited

Loading

South-Paw commented Sep 2, 2018 •

edited

Loading

South-Paw commented Sep 8, 2018 •

edited

Loading

Saving image files to `data/img` folder (after resizing)

South-Paw commented Sep 8, 2018 •

edited

Loading

South-Paw commented Sep 10, 2018 •

edited

Loading

andersbrandt commented Sep 15, 2018 •

edited

Loading

South-Paw commented Sep 17, 2018 •

edited

Loading

resir014 commented Sep 19, 2018 •

edited

Loading

South-Paw commented Sep 20, 2018 •

edited

Loading