Item demographics test and gun appearance rate rebalancing.

[kevingranade]

Summary

Infrastructure "Use test enumerating item spawns to make assertions about the rate of appearance of different items."

Purpose of change

In support of #37571 this adds a test to repeatedly run mapgen and tally the items generated so we can make some assertions about their distributions.

Describe the solution

This extends the existing overmap terrain coverage test to use the counts of oter_types it already aggregates to drive repeated mapgen of the oter_types and enumerate the items spawned.
Specifically it tallies up the guns and ammo enumerated in item_demographics.json and then makes assertions about their relative proportions as established in #37571
Unfortunately our mapgen and item spawning is very complex and has many rare events, which means that in order to even approach getting usable data, I had to run mapgen for about an hour and a half. Clearly this isn't a candidate for running in the test suite, so the test is disabled with a bool, which specifically disables the mapgen sampling and then skips making assertions about the aggregated data. In addition, even if we're willing to tack another hour and a half to test run time, it's still too noisy to make meaningful assertions, it has occasional outliers that would cause constant failures.
So instead, we can run this periodically as an audit step to tune up item spawns. Possibly future improvements will be able to address these issues.
Once I got ok results I started iteratively adjusting item groups, special spawns, extras, etc as seemed appropriate to nudge appearance rates toward the desired goal.
One of the really core changes is in data/json/itemgroups/Weapons_Mods_Ammo/guns.json and data/json/itemgroups/Weapons_Mods_Ammo/guns_by_calibre.json
The very frequently used guns_common, pistols_common, rifles_rare, etc item groups have been mostly overhauled to reference a new set of item groups named like pistols_common_9mm, pistols_common_22 etc, which enumerate the matching guns in those calibres and categories. This allows new guns to be added or removed, or have their rate of appearance (relative to other guns of the same calibre) changed without disrupting overall spawn rates of guns of a similar type.
I also replaced a number of places that apwn specific lists of guns with references to these item groups instead.
I'll comment in the diff for specific changes, but in general, the distributions of guns was way too "flat", in that common guns, rare guns, pistols/rifles/shotguns, etc were spawning with very similar weights, resulting in lots of should-be-very-rare guns spawning in very large quantities.

This is what my final test run looked like:

Calibre	Expected	Actual	Difference
10mm	0.00103f	0.00297f	0.0019363
123ln	0.00103f	0.00061f	-0.000416743
300blk	0.00206f	0.00248f	0.00042012
270win	0.00309f	0.00532f	0.00223532
410shot	0.00309f	0.00202f	-0.00106332
762R	0.00412f	0.00613f	0.00201301
3006	0.00823f	0.01265f	0.00441728
45colt	0.00823f	0.00544f	-0.00278693
44	0.01543f	0.00932f	-0.00610817
38	0.01955f	0.03053f	0.010983
308	0.02572f	0.03193f	0.00621175
shot	0.02675f	0.02643f	-0.000322772
357mag	0.03498f	0.03312f	-0.00186342
40	0.04115f	0.05048f	0.00933166
45	0.05041f	0.0524f	0.0019917
other	0.11728f	0.12314f	0.00585767
223	0.1286f	0.09812f	-0.0304818
22	0.20988f	0.21636f	0.0064837
9mm	0.21502f	0.20663f	-0.00839059

Expected and actual are in proportion of global gun spawns, e.g. 9mm is expected to be 21.5% of gun spawns, it was actually about 20.6% in this test run, which means it's 0.08% lower than intended. Considering the amount of variation in mapgen, extras etc, I'm actually pretty ok with some of these being about 1% off. I'm less happy about .223 in particular being about 3% low, but that would also represent at least several more days of tuning to try and get it within 1% or so and I just don't have it in me right now.

Now what does this look like in terms of actual impact?
Shotguns are much, much rarer now.
Pistols are much, much more common.
9mm, .22, and .223/5.56 are now staggeringly dominant, and everything else is commensurately rarer.
,40, .45, and .357 aren't THAT rare, but most everything else is.
Rifles other than .22 and .223 in particular make up less than 5% of total gun spawns now.

Describe alternatives you've considered

Not caring! (nope)
Some kind of save analysis tool? (pretty prohibitive to make a map that big I think? Lots of wasted effort.
Instead of running mapgen to completion and tallying the results, it might be possible to instead instrument the item spawning code to enumerate the rates of appearance of matching items per item group, then hook into mapgen and detect what item groups are being invoked, then combine those rates of appearance to get overall spawn rates. This would involve hooking into a ton of different subsystems and be subject to breakage every time one of them changes.

Testing

Ran it repeatedly for days and days and tweaked things until I got the results relatively stable.
If you can think of a way to validate the results without building one of the above solutions (or doing so very quickly?) I'm all ears.

Additional context

This replaces #69169, which was auto-closed, then I force pushed to this branch which locked the PR? Instead of trying to sort that mess out I just pushed a new PR.
I was originally hoping to cover all of guns and magazines and ammo, but this has literally drug on for months already and I have ok ish coverage for guns, I don't have it in me to launch into a whole new batch of overhauls.

[RanikOrtega]

Now what does this look like in terms of actual impact? Shotguns are much, much rarer now. Pistols are much, much more common. 9mm, .22, and .223/5.56 are now staggeringly dominant, and everything else is commensurately rarer. ,40, .45, and .357 aren't THAT rare, but most everything else is. Rifles other than .22 and .223 in particular make up less than 5% of total gun spawns now.

While I understand the point of auditing. If this is auditing weapon spawns is this pushing too far into unnecessary realism based off of real world numbers rather than fun gameplay in a fictional setting? This also reduces game content and fun for finding unique things. "ooh a .308 or a .50 BMG"

While this mostly addresses civilian spawns at some point it is a game and does not need to be based off of real world numbers.

[kevingranade]

Limiting access to better and more varied guns is something I would want to do from a gameplay point of view even if my solution wasn't based on IRL numbers. Most games do this by there only being a tiny sampling of items in the game in the first place, and delaying the appearance of better items until after the player achieves certain levels of progression. We're doing it by there being a huge number of items, but only a small fraction of them are "interesting", so it requires searching to find better ones (unless you get lucky, that's also an option).

This also reduces game content and fun for finding unique things. "ooh a .308 or a .50 BMG"

It does not, you can still do it, it's just likely to take you longer. The opposite is a big problem as well, "welp found an assault rifle and an anti-materiel rifle on day 3, never need to find another gun".

While this mostly addresses civilian spawns at some point it is a game and does not need to be based off of real world numbers.

This is nonsense considering this is and has for about a decade been how the game is developed.

[RanikOrtega]

This is nonsense considering this is and has for about a decade been how the game is developed.

Again it is a game. And at some point some amount of handwaving occurs. Zombies are not realistic, mutations are not realistic and interdimensional travel is not realistic. 🤣 I'm just saying some small amount of gun caliber inclusion rather than 100% strict adherence to realism attempts for the purpose of gameplay is not nonsense.

If you wanted to say it was a game balance issue of damage potential per caliber that's totally understandable to me but not a 100% adherence to best guess Google Analytics stats.

[kevingranade]

I don't follow what you're saying at all, you can still find them. They have not been removed from the game, it's just that 9mm is now meaningfully more common than say 10mm instead of most calibers, no matter how obscure appearing in similar numbers.

[KnifeFightBears]

So if 0.21502f is supposed to represent 21% of gun spawns, 0.00103f is supposed to represent .1%?

This ratio seems wild for a comparison like 9mm to 10mm where one is obviously more common than the other, but how do you figure the rarity of .270 in a part of the country where half the states don't have easy access to handguns but sell rifles in hunting calibers at every sporting goods store?

Also, shotguns are 1/10th as common as a 9mm handgun? Is there information to support this? Seems like interesting trivia if it's a real number

[kevingranade]

See #37571 for sources.

[RanikOrtega]

See #37571 for sources.

That's data from one online vendor from 2012 - 2014. Not including gun stores, local sport stores, Big name Sports Stores like Academy, Walmart (Which still has guns) or other online vendors. Nor representing trends of the past ten years. I cannot say that is a highly comprehensive list to be making major changes.

[KnifeFightBears]

See #37571 for sources.

That's data from one online vendor from 2014. Not including local sport stores, Walmart or other online vendors. Nor representing trends of the past ten years. I cannot say that is a highly complete list to be making major changes.

This seems like a really weird way to do these analytics, to be honest

2012-2016 the gun market was going crazy because everyone was expecting Obama to legislate, this had a huge impact on the sales of ammo in classes that people were anticipating restrictions, namely common handgun calibers and 5.56; Additionally, our significant hunting culture buys a lot of ammo from brick and mortar outlets, which is why all of the feed lot stores in my area have three aisles of ammo (about a third of which is shotgun shells alone)

It doesn't surprise me that shotshells are a small percentage of online sales (they're heavy to ship and commonly available at gas stations in rural communities) but Pew says half of american gun owning households have a shotgun, and they represent about 16% of the share of single gun households

https://www.pewresearch.org/social-trends/2017/06/22/the-demographics-of-gun-ownership/

I don't mean to be a complaining carl for what it's worth, I get what you guys are doing with this particular list and understand, I think hunting culture gets the short end of the stick a lot with these PRs though as a result of how you're doing the numbers. I don't have a good suggestion for how to do it better, I just wanted to say it stuck out to me.

[RanikOrtega]

See #37571 for sources.

That's data from one online vendor from 2014. Not including local sport stores, Walmart or other online vendors. Nor representing trends of the past ten years. I cannot say that is a highly complete list to be making major changes.

This seems like a really weird way to do these analytics, to be honest

2012-2016 the gun market was going crazy because everyone was expecting Obama to legislate, this had a huge impact on the sales of ammo in classes that people were anticipating restrictions, namely common handgun calibers and 5.56; Additionally, our significant hunting culture buys a lot of ammo from brick and mortar outlets, which is why all of the feed lot stores in my area have three aisles of ammo (about a third of which is shotgun shells alone)

It doesn't surprise me that shotshells are a small percentage of online sales (they're heavy to ship and commonly available at gas stations in rural communities) but Pew says half of american gun owning households have a shotgun, and they represent about 16% of the share of single gun households

https://www.pewresearch.org/social-trends/2017/06/22/the-demographics-of-gun-ownership/

I don't mean to be a complaining carl for what it's worth, I get what you guys are doing with this particular list and understand, I think hunting culture gets the short end of the stick a lot with these PRs though as a result of how you're doing the numbers. I don't have a good suggestion for how to do it better, I just wanted to say it stuck out to me.

In my relatively small city we have at least 5 gun stores that I know of. They have been around for decades and are not hurting for business. The type of people who want to buy lots of ammo are not always going to be doing so online (as they are distrustful of tracking) but will instead drop hundreds on ammo right at the store or even the indoor gun ranges themselves which also sell ammo.

Not to mention that in the past few years "newer" rounds like .300 blackout only started in 2010 and may have become more popular after the fact since it was only 2-4 in the 2012-2014 time frame.
https://en.wikipedia.org/wiki/.300_AAC_Blackout

Or 6.5 Creedmore which came out in 2008.
https://en.wikipedia.org/wiki/6.5mm_Creedmoor

Also see https://backfire.tv/popular-cartridges/

[kevingranade]

Give me better data and I can see about incorporating it. Anecdotes aren't helpful.
Out of date is fine, "my favorite new caliber is underrepresented" is fine, the status quo of "everything is about the same I guess" is not.

Edit to address this:

This seems like a really weird way to do these analytics, to be honest

It absolutely is a weird backwards way to do it yes, but it's the only credible data I've found, period. The only thing that comes close is writing a script to scrape gunbroker.com for their listing history, which is similarly biased, just in a different direction.

I read the pew research article, the problem is it's just not particularly actionable for counts of guns.
I'm summarizing and normalizing the reported proportions here to make things more straightforward to discuss.
All percentages are percentages of the total population. The article mixes up global and relative percentages a ton.
30% of adult respondents said they own at least one gun.
10% said they own just one gun.
6.2% own a pistol.
2.2% own a rifle.
1.6% own a shotgun.
20% said they own more than one gun.
There's no breakdown of which guns this population owns.
about 12% own 2-4 guns.
about 10% own 5+ guns.

Correlation indicates that among people who own more than one gun, they have more than one reason for owning it. i.e. "self defence" + "hunting" + "target shooting" etc, which suggests that they have a broad selection of gun types.
What does this tell us about absolute ratios? Practically nothing.

[fairyarmadillo]

This also reduces game content and fun for finding unique things. "ooh a .308 or a .50 BMG"

This is a tangent and won't affect the conversation about the data, but it could be an incentive to add spawns to specific locations where less-common gun types would make sense. Shotguns to the hunting lodge, sniper rifles to military locations, etc. Even if these items are vanishingly rare in the general loot pools, players could potentially seek them out in places they'd be more likely to appear.

Martial arts books are rare among books, but common in dojos, to use an example. This creates organic gameplay goals as players try to reach locations more likely to have the stuff they want rather than just looking for needles in haystacks.

[RanikOrtega]

This also reduces game content and fun for finding unique things. "ooh a .308 or a .50 BMG"

This is a tangent and won't affect the conversation about the data, but it could be an incentive to add spawns to specific locations where less-common gun types would make sense. Shotguns to the hunting lodge, sniper rifles to military locations, etc. Even if these items are vanishingly rare in the general loot pools, players could potentially seek them out in places they'd be more likely to appear.

Martial arts books are rare among books, but common in dojos, to use an example. This creates organic gameplay goals as players try to reach locations more likely to have the stuff they want rather than just looking for needles in haystacks.

I could see that as a reasonable compromise. Certain things are not necessarily in a common home defense category but are extremely prevalent in hunting. 12 gauge, .308, .300 winmag etc etc and then more modern hunting rounds that have become their own market in recent years.

[kevingranade]

That's less a compromise and more how things already work. The major "just spawn me a gun" item groups are very, very low on shotguns (about 1% shotguns, and obviously that doesn't include any locations where they're spawning in a drawer or a gun safe), and most of the places they spawn are asking specifically for a shotgun, like police armories, vehicle trunks, or hunting-themed locations.