Indexer Core v4.0

This is an official release of the v4 Indexer Core that has been in production for a couple weeks at exiletools.com. It includes a massive re-factor of the pipeline via asynchronous Kafka queue processing.

Version 3 - Entirely new system using the Stash Tab API

Note: These Release Notes are for "official" releases but you should almost always run on the current master branch to get the latest changes. I don't make a full release for every little bug fix.

General Changes:

Everything is different in the pipeline. There's no more database in the interim, there's no more need to set up various jobs for the pipeline, etc. There is now a single job that runs continuously to consume the river made available by the official Stash Tab API and insert this data directly into Elastic Search.

Please review the updated README to understand how this works. All of the abandoned older code is temporarily moved to .old-files-from-v2 in case they need to be referenced.

Pipeline

• There is an option to consumed API data for reconstruction
  • If enabled, each change_id is saved to its own file, allowing the index to be rebuilt on demand.
  • I haven't yet written a tool to do this
• There is an option to log every file indexed as a single line of JSON into a JSON log
  • Intended to ultimately be used for web sockets and filtering before items even hit the index

Tools

• A new tool called watch-modified-items.pl has been added. This will run forever and confirm items are going into the index. Handy for debugging, but should be ctrl-c'd when you're done debugging.
• pull-item-data-from-poedb.pl has been moved in from my personal tool chest. It's just a simple script to create the gear base type hash. If a bunch of new stuff gets added in Ascendancy you may need to run it yourself or just wait for me to push an update.
• extract-json-from-thread.pl has been changed a bit to work with the new subroutines
• setup scripts are now in tools\setup

WWW directory

Bunch of changes here:

• www/html/docs now contains the documentation data that you can see at http://api.exiletools.com/info
• helloworld.html and js/es-helloworld.js contains a very simple AngularJS example to get up and running against the public ES API
• shop-watcher.html and js/es-watcher.js contain code to allow users to monitor a seller account in the index for changes in near-realtime
• Some other random junk in there for skilltree stuff that are works in progress

Mapping / Document Information Changes in v3:

There should be no major breaking changes in v3, however there are some minor breaking changes because of the new Stash Tab API - some shop forum related fields are obviously gone now.

There are also a bunch of new fields available as well as some various other changes shown below:

• New shop fields have been added for the Public Stash Tab API. These include:
  • shop.note : Contains the original note field, unedited
  • shop.priceSource : Indicates whether the price (if any) was determined from the note or stash tab title
  • shop.defaultMessage : A simple default message that can be copy/pasted to a seller (you can of course build your own and ignore this)
  • shop.stash : A new object with API level stash metadata
  • shop.stash.stashID : A UUID identifying the stash used by the API
  • shop.stash.inventoryID : The internal GGG inventory ID of the stash, I don't think this is used anywhere
  • shop.stash.stashName : The original name of the stash (note: UTF8 is supported! This field can get fuuunky)
  • shop.stash.xLocation : The X coordinate of the item in the stash
  • shop.stash.yLocation : The Y coordinate of the item in the stash
  • shop.lastCharacterName : The name of the character logged in when the stash tab was updated
• Some fields have been removed as they are no longer relevant. These include:
  • md5sum
  • shop.threadid
  • shop.threadTitle
  • shop.sellerIGN
  • shop.lastUpdateDB
• Added new modsPseudo calculations to create the following new pseudo mods:
  • #% Total increased Evasion Rating
  • #% Total increased Armour
  • #% Total increased Energy Shield
  • #25
• Added attributes.baseItemName which will contain the determined underlying base item type (i.e. "Rustic Sash", "Spiny Maul", etc.)
  • Updated all gear base types to include all currently known items
  • #96
• Added new Estimated 20% Quality calculations for Armour, Evasion, Energy Shield, and Physical DPS. The nature of the rounding in these calculations means it may be off by 1-2pts in some situations from the actual in-game result. These new fields are part of a new propertiesPseudo object as follows:
  • pseudoProperties.Armour.estimatedQ20.Evasion Rating
  • pseudoProperties.Armour.estimatedQ20.Armour
  • pseudoProperties.Armour.estimatedQ20.Energy Shield
  • pseudoProperties.Weapon.estimatedQ20.Physical DPS
  • #25
• A new sockets information field called sockets.allSocketsGGG will be generated that includes the socket list in GGG's text style layout (i.e. G-G B-B means two linked green sockets followed by two linked blue sockets)
  • #99
• Any field which has min and max attributes should now also have an avg attribute (which is simply (min + max) / 2 for search purposes
  • #102
• Ascendancy Enchantments have their own object section in the JSON data. As these are always fixed values, they will all be entered in to the mapping as boolean true objects under enchantMods - note that any apostrophes in the mod name will be removed. Additionally, the attributes.enchantModsCount value shows how many lines of enchantMods there are (presumably this will never be over 1, but..)
  • #100
• Item level is now available in the JSON data from GGG!
  • attributes.ilvl
  • #101
• Gem "Cast Time" and "Cooldown Time" (and any other Gem property ending in " sec" will now have " sec" removed and be stored as a number
• Items with Stack Sizes will now set properties.stackSize.current and properties.stackSize.max
• There is now an attributes.weaponType property which indicates a weapon's type in a searchable field (i.e. attributes.weaponType : Dagger) so you do not have to search for properties.Weapon.type.Dagger : true
  • #105
• There is now a boolean shop.hasPrice field which will be set to true if the item has a price and false if it does not
  • #104
• Socket counts for different colors are now added under sockets, these include:
  • sockets.totalWhite
  • sockets.totalRed
  • sockets.totalBlue
  • sockets.totalGreen
  • #106

v2.4.1 - Critical bug fix!

This release includes a critical bug fix to spoiler price detection - prior to this release, prices in spoiler titles had periods removed as meta characters, thus creating prices of "175 exalt" instead of "1.75 exalt"

If you have an active index for this release, you will need to rebuild the items database from disk to correct these prices! This can be done by clearing the items table (truncate it) completely, then changing all items in web-post-track to a processed of 0:

update `web-post-track` set `processed`=0 where `processed`>0;

You will then need to run load-data-from-disk.pl to reload all items into the DB, then load those items into Elastic Search.

v2.4 - Bunch of fixes for Talisman +more

This release includes a ton of minor bug fixes for Talisman, as well as the beginnings of integration the Reports Dashboard and Price API/Macro data into the main project (those are both still works in progress).

This is a breaking update. You must destroy and recreate the index, then re-import all items to deploy this update!

• Talisman types are now detected correctly
• chaosEquiv has been updated to 80:1 for Exalted Orbs
• Elemental damage mods are now calculated correctly as Fire Damage etc., and Elemental DPS and Total DPS now accurately reflects all Elemental Damage
• The threshold jewels that have broken mods (newlines for no reason) are detected as a single mod now
• Various properties are now correctly flagged as numbers instead of strings
• The type of sale (b/o fixed etc.) is now indicated in the index

Plus various other changes

v2.3 - Huge mapping overhaul and bug fixes for 2.1

This release includes a large number of mapping changes in preparation for Path of Exile 2.1. The Issues for the Indexer for milestone Talisman Release have more information, however a short recap includes:

• A number of mods have been removed from modsTotal including:
  • All boolean mods
  • All cosmetic mods
  • All Flask, Gem, Map, and Divination Card mods
• Gem types will be properly detected for single type gems and added to properties.Gem.type
• Weapon types should be properly detected and added to properties.Weapon.type
• Map properties will now be numbers and the leading + from Quantity, Rarity, etc. have been removed
• Elemental, Chaos, and Total DPS are now calculated properly for Weapons
• Pseudo Mods now have normal human readable names
• Support has been added for Talisman items as a new equipType (in theory)
• Items with Unknown League will not be added to the index
• Items with an Unknown baseItemType will not be processed for mods, preventing mods.Unknown mappings/etc. from being created
• Improper base item type detection has been fixed, so items like Saintly Chainmail will no longer be identified as a Sai
• Added specific forum information to runs and threads so that forums can be refreshed separately

Installation

This requires a completely rebuild of the pipeline to run properly, including:

• You must truncate all tables in indexer except for league-list - note that thread-last-update has a new column.
• Upgrade ElasticSearch to 2.1 and re-create both the item and stats indexes in ElasticSearch
• Wait until 2.1 is released before resuming indexing against the forums to ensure the new mappings match the new items

v2.2.1 - Mapping bugfix

This release contains a bug fix for the ElasticSearch mapping of the indexing-stats table. The mapping was accidentally set to use epoch-millis for timestamp fields instead of epoch-seconds. Users who upgraded will need to run elasticdump (or similar) to dump out the indexing-stats table then re-create it with the epoch-seconds mapping and re-import.

Version 2.2, Statistics now in ElasticSearch instead of Mariadb and Config File changes

This release includes two breaking changes that require updating configuration files and creating new indexes:

1. The config file now uses camelCase and will need to be updated (Issue #43)
2. Statistics about indexing runs and thread history are now stored in ElasticSearch instead of MariaDB. This will allow better/faster/easier graphing and data analysis on the health and indexer activity. (Issues #40 and #41)
   • setup/es-recreate-stats-index.pl will allow you to test and create these indexes
   • New config lines are required for these
   • tools/format-fetch-stats-and-load-to-es.pl will transition current fetch-stats items into ES
   • tools/format-thread-update-history-and-load-to-es.pl will transition current thread-update-history items into ES

It also includes a fix for Removed threads constantly trying to be refreshed by refresh-forum-threads.

For new installation instructions, see previous version release notes.

Version 2.1, Minor Bugfixes and Feature Enhancements

Version 2.1 includes some minor bugfixes and new support in the database and ElasticSearch for shop.saleType allowing you to search/filter/display based on b/o, price, offer, or c/o in rare circumstances.

Version 2.0 is a stable tested release of the Indexer as currently (2015/11/19) used by http://exiletools.com - most of the major mapping problems with things like Divination Cards have been fixed, and the thread fetching, data analysis, trending, and dynamic mapping into Elastic Search have all been stabilized and made reliable.

This version works with the current pathofexile.com shop forums. Active leagues at the time of writing for the indexer are as follows:

MariaDB [indexer]> select * from `league-list` where `active`>0;
+--------------+---------------------+-----------------------+------------+------------+--------+-----------------------+----------------+------------------------------------------------------+----------------------------------------------+-------------+
| league       | prettyName          | apiName               | startTime  | endTime    | active | itemjsonName          | archivedLadder | shopForumURL                                         | shopURL                                      | shopForumID |
+--------------+---------------------+-----------------------+------------+------------+--------+-----------------------+----------------+------------------------------------------------------+----------------------------------------------+-------------+
| darkshrine   | Darkshrine          | Darkshrine (IC003)    | 1446490800 | 1449518400 |      1 | Darkshrine (IC003)    |              0 | http://www.pathofexile.com/forum/view-forum/597/page | http://www.pathofexile.com/forum/view-thread | 597         |
| darkshrinehc | Darkshrine Hardcore | Darkshrine HC (IC004) | 1446490800 | 1449518400 |      1 | Darkshrine HC (IC004) |              0 | http://www.pathofexile.com/forum/view-forum/598/page | http://www.pathofexile.com/forum/view-thread | 598         |
| hardcore     | Hardcore            | Hardcore              |       NULL | 2147483647 |      1 | Hardcore              |              0 | http://www.pathofexile.com/forum/view-forum/305/page | http://www.pathofexile.com/forum/view-thread | 305         |
| standard     | Standard            | Standard              |       NULL | 2147483647 |      1 | Standard              |              0 | http://www.pathofexile.com/forum/view-forum/306/page | http://www.pathofexile.com/forum/view-thread | 306         |
+--------------+---------------------+-----------------------+------------+------------+--------+-----------------------+----------------+------------------------------------------------------+----------------------------------------------+-------------+

To install and run your own version of the indexer, you will need to:

1. Install MariaDB or MySQL
2. Import the indexer table structure from setup
3. Install Elastic Search 2.0
4. Create the default elastic search tables using es-recreate from setup
5. Install Perl and all associated Perl modules called in various Require lines
6. Update the config and dbcreds files with the database and ES host information, credentials, etc.
7. Populate the league-list with the shops you want to index
8. Start running get-forum-threads.pl and follow the pipeline from there!

Of course, you can always simply access the ExileTools public index instead of rolling your own:

http://api.exiletools.com/info

Cheers!

-Pete

Initial Open Source Release

Version 2.0 is a stable tested release of the Indexer as currently (2015/11/19) used by http://exiletools.com - most of the major mapping problems with things like Divination Cards have been fixed, and the thread fetching, data analysis, trending, and dynamic mapping into Elastic Search have all been stabilized and made reliable.

This version works with the current pathofexile.com shop forums. Active leagues at the time of writing for the indexer are as follows:

MariaDB [indexer]> select * from `league-list` where `active`>0;
+--------------+---------------------+-----------------------+------------+------------+--------+-----------------------+----------------+------------------------------------------------------+----------------------------------------------+-------------+
| league       | prettyName          | apiName               | startTime  | endTime    | active | itemjsonName          | archivedLadder | shopForumURL                                         | shopURL                                      | shopForumID |
+--------------+---------------------+-----------------------+------------+------------+--------+-----------------------+----------------+------------------------------------------------------+----------------------------------------------+-------------+
| darkshrine   | Darkshrine          | Darkshrine (IC003)    | 1446490800 | 1449518400 |      1 | Darkshrine (IC003)    |              0 | http://www.pathofexile.com/forum/view-forum/597/page | http://www.pathofexile.com/forum/view-thread | 597         |
| darkshrinehc | Darkshrine Hardcore | Darkshrine HC (IC004) | 1446490800 | 1449518400 |      1 | Darkshrine HC (IC004) |              0 | http://www.pathofexile.com/forum/view-forum/598/page | http://www.pathofexile.com/forum/view-thread | 598         |
| hardcore     | Hardcore            | Hardcore              |       NULL | 2147483647 |      1 | Hardcore              |              0 | http://www.pathofexile.com/forum/view-forum/305/page | http://www.pathofexile.com/forum/view-thread | 305         |
| standard     | Standard            | Standard              |       NULL | 2147483647 |      1 | Standard              |              0 | http://www.pathofexile.com/forum/view-forum/306/page | http://www.pathofexile.com/forum/view-thread | 306         |
+--------------+---------------------+-----------------------+------------+------------+--------+-----------------------+----------------+------------------------------------------------------+----------------------------------------------+-------------+

To install and run your own version of the indexer, you will need to:

1. Install MariaDB or MySQL
2. Import the indexer table structure from setup
3. Install Elastic Search 2.0
4. Create the default elastic search tables using es-recreate from setup
5. Install Perl and all associated Perl modules called in various Require lines
6. Update the config and dbcreds files with the database and ES host information, credentials, etc.
7. Populate the league-list with the shops you want to index
8. Start running get-forum-threads.pl and follow the pipeline from there!

Of course, you can always simply access the ExileTools public index instead of rolling your own:

http://api.exiletools.com/info

Cheers!

-Pete