creature_loot_template python scraper

[Deadlineem]

Moved to #31339

[mdX7]

Sadly loot isnt as easy as just grabbing wowhead data.
Wowhead data usually is tainted due to several factors. E.g. it occassionally also contains pserver data. Or AoE loot adding items from a creature in loot range to the wrong one (there are multiple cases where e.g. a quest item dropped only by specific creature are in other creatures loot tables. (Thats a limitation of their addon)
The items are usually grouped (especially for open world mobs and dungeon trash). And depending on some factors the creatures also share loot groups. So even with scraping tools you'd still have to manually review every case for proper loot pooling.
Boss loot is usually way easier to handle.
And yes, there are other tools like this, usually without taking the grouping into account. With people giving us the output of thousands of creatures "go merge it, my tool is powerful, you dont have to review it, trust me bro".

[Deadlineem]

Sadly loot isnt as easy as just grabbing wowhead data. Wowhead data usually is tainted due to several factors. E.g. it occassionally also contains pserver data. Or AoE loot adding items from a creature in loot range to the wrong one (there are multiple cases where e.g. a quest item dropped only by specific creature are in other creatures loot tables. (Thats a limitation of their addon) The items are usually grouped (especially for open world mobs and dungeon trash). And depending on some factors the creatures also share loot groups. So even with scraping tools you'd still have to manually review every case for proper loot pooling. Boss loot is usually way easier to handle. And yes, there are other tools like this, usually without taking the grouping into account. With people giving us the output of thousands of creatures "go merge it, my tool is powerful, you dont have to review it, trust me bro".

I'm 1000% sure it needs review, however, for specific NPC's (https://www.wowhead.com/npc=426/redridge-brute#drops) like this one, it has 329 drops listed, This scraper shows 331 items added so it seems, for the most part, pretty accurate in terms of that which would make single NPC's drops output perfectly or NPC's in Dungeons (lets say ID 4000-5000 is deadmines NPC's for example, though its not that easy) groups of NPC drop loot. I have yet to test the SQL file this python script creates, im letting it run on 240,000 ID's but i can say its pretty darn accurate so far cross referencing some of the data in the browser.

Stormwind city guard has no loot, so it puts all the stormwind city gear (imperial helm, etc,) as their drop loot for some reason so I am aware its even giving NPC's with no loot, some loot.

from ID 1 to ID 240,000 (skipping those with empty results in between) its estimated at 34 hours to output all the data. I encourage you to test this on specific NPC's like the ones in deadmines (.npc info on them for the ID and cross ref in wowhead first) that currently have no loot, generate the SQL for that creature and import it, see if the loot comes up.

EDIT: This is a prototype script btw, even if all item data is correct, all the drop rates default to 0, this is something i would like to change but the json data from wowhead does not seem to contain these %'s even though they display on their website, I plan to tweak this to get all accurate drop rate %'s for each item as well, as much real data as possible to keep the hand work down to absolute minimum.

I am also looking to expand on this, group ID's and ItemTypes (0 or 1) are for sure things I would like to address and learn more about. Proper Min/Max Counts to be added (1/1 is better than no drops at all imo)

NOTE: This python script supplied was build for and is running on a CLI Ubuntu 22.04 VPS currently, no idea how it works on windows.

[Deadlineem]

It seems GroupID's are based off of how you interact with said item, for example, group ID 3 is used for linen cloth and refreshing springwater, you dont directly interact with the cloth, however it is used when doing a crafting interaction for something like bandages, while the water is directly drinkable when interacting with the item. -- Enlighten me if im wrong.

[mdX7]

The groups i talked about are more about creature types (e.g. elementals, same level) usually drop the same grey items, sometimes different green items, sometimes the same, [...]
According to some people the gear dropped is also individually grouped too. But i didnt verify that yet. Currently we only group by "color" and check for common stuff to reuse groups.
Example commit being: 93f3b60

[Deadlineem]

The groups i talked about are more about creature types (e.g. elementals, same level) usually drop the same grey items, sometimes different green items, sometimes the same, [...] According to some people the gear dropped is also individually grouped too. But i didnt verify that yet. Currently we only group by "color" and check for common stuff to reuse groups. Example commit being: 93f3b60

When adding creature loot, do references also need to be added with them to match?

[Naddley]

You create references for mostly all "groups" of loot like: junk is one reference, greens are one reference and so on. This will cover to not duplicate everything since blizz re-use this references. If you look at my commits you will see that most of the different Ravens for example share the same Junk reference.

[CraftedRO]

I just wonder why almost everyone think we have python installed and ready to run their script anytime, I mean is not like is a batch/shell script.

[mdX7]

I just wonder why almost everyone think we have python installed and ready to run their script anytime, I mean is not like is a batch/shell script.

People develop their tools in whatever language they like, its not defined by law which language to use.
If you want to use it - install Python, if you dont want to use Python, you cant run the tool. But you may translate it to other languages as needed.
I'm also usually writing everything in Python as its very good to write stuff quickly (as in writing quickly, not processing quickly). Also crossplattform due to being interpreted, so thats a big plus aswell. And for stuff which does not require fast performance (like this case) its totally legit to write it in Python.

tl;dr
Feel free to translate to other languages so you dont have to install Python :P

EDIT: Also keep in mind Python is pretty much """preinstalled""" with every linux distro

[CraftedRO]

ok good to know, the question was not mean to offense was just a curiosity.

Feel free to translate to other languages so you dont have to install Python :P

I will actually do this but modify to scrap some other data I need, I was having in plan to do smth like.

EDIT: Also keep in mind Python is pretty much """preinstalled""" with every linux distro

yeah but last time I checked Windows is still the leader in top usage share os.

[Deadlineem]

ok good to know, the question was not mean to offense was just a curiosity.

Feel free to translate to other languages so you dont have to install Python :P

I will actually do this but modify to scrap some other data I need, I was having in plan to do smth like.

EDIT: Also keep in mind Python is pretty much """preinstalled""" with every linux distro

yeah but last time I checked Windows is still the leader in top usage share os.

I run my TrinityCore 11.2.x server on https://wowspire.fpr.net/ this server runs off Ubuntu 22.04 CLI (no GUI) and automatically updates (using a bash script) every 12 hours from this repository directly. This is also where this script is running. I don't really ever prefer to install python on my Windows PC but its really great for CLI linux. The main thing this lacks is because there is no GUI that opening the browser has to be headless, therefore its more difficult to parse HTML from the page, but parsing json data (which is how this works currently) works pretty well. If this was GUI based I would be able to do everything in it including parse drop %'s, ensure the data is ONLY retrieved from tab-drops (wowheads drops tab html id) and possibly even categorize by color as mentioned above.

I just let this script run on a full 1-240,000 cycle, im going to test just how accurate it is with a full import to my creature_loot_template table. If this breaks, fixing it for me is literally as simple as deleting the world table and restarting my server. MY consensus to the idea was, I would rather import a massive list of 90% completed drops (that actually work instead of 60% of the world mobs not dropping anything) with drop rates set to 25 for everything vs not having that 60% of data at all. Im sure with the proper tweaks, this script could benefit TrinityCore heavily and automate some tedious tasks (especially since there are basically 240k NPC ID's, I would off myself doing each one by hand) like creature_loot_template, reference_loot_template, item_loot_template, etc.

If its built fully and correctly, this script could potentially populate ALL of that data for the developers allowing them to focus more on the core itself vs having to manually add entries or take a chance at populating it from the game or another source. I did test this with deadmines last night (went to wowhead, copied all deadmines NPC ID's) and the results were as follows:

Prior to the import, no loot dropped at all in the dungeon from any mobs.

After the import, everything AFTER helix gearbreakers room dropped all the items they should drop, however, everything in/before his room dropped nothing, so its about a 50% success here so far (script defaults the drop rate to 25 for testing purposes)

I've also tweaked it more since last update, allowing more custom args

python3 scrape_wowhead.py --help

Usage: scrape_wowhead.py [options] [npc_id|start end|id1,id2,id3]

Options:
  --name <filename>    Output SQL file name (default: creature_loot_template.sql)
  --chance <value>     Force chance % for all items (0–100)
  --min <value>        Force MinCount for all items (>=1)
  --max <value>        Force MaxCount for all items (>=1)
  --help               Show this help message

[Aokromes]

well, a tool doesn't belong to BUG tracker. put it into discusions.