-
Notifications
You must be signed in to change notification settings - Fork 0
Home
IR processing is influenced by many factors , first of all it’d depend on wakeup interval of IR process since you have added ir_asynchronous: 1 to sm.ini so that instead of every session directly updating IRindexes as data is updated, the IR index changes are logged to an internal table (irqueue), and a background process performs all the IR index updates separately. Also, DB server performance plays crucial role at this point of time apart from this we also need to understand CPU & memory utilization of SM server.
We can monitor the IR Thread performance using sysinternal tools provided by Microsoft.
Generally, For very large environments with huge number records in each table, and where each record has a significant amount of data of type array, it may be advisable to remove the array type fields from the IR keys (in the dbdict). This change will trigger an IR regen and test again to determine how long this would take. The amount of data that gets indexed can be rather large and as a result may severely impact performance.
For example : You may remove update.action and brief.description from probsummary dbdict
Also, Using and maintaining the stop words list is highly recommended to make sure that IR queries and regens are most efficient.
To determine which terms should be added to the stop words file, run the vrir command from sm –util occasionally (may be every two months).
Any term that occurs more than 10,000 times should be added to the stop words files. The IR system does not function reliably with terms that occur >65,000 times.
D:\HP\Service Manager 9.30.511\Server\RUN>sm -util
add: add cls: close cnt: count del: delete dis: display fnd: find get: get nxt: next opn: open pat: patch prv: previous qbe: qbe reg: ir regen rst: reset rmv: remove upd: update vrir: verify IR x: EXIT
Enter your choice: vrir
Validate an IR file (Version: 9.30.511 Build: 511) [03/19/2012 19:14:25]
Enter fully qualified name for the IR file: ir.probsummary
IR validation succeeded. See log for details
add: add cls: close cnt: count del: delete dis: display fnd: find get: get nxt: next opn: open pat: patch prv: previous qbe: qbe reg: ir regen rst: reset rmv: remove upd: update vrir: verify IR
x: EXIT
You can filter the stop words to be added in english.stp from the log file generated to further tune performance.
Hope this clarifies potential causes behind slow performance of IR.
The procedure required IR updates being done asynchronously as it can be stopped and queued. Therefore, ir_asynchronous:1 MUST be on in the production sm.ini file.
The irqueue dbdict is a queue of IR updates not yet being placed into the IR index. So this is a point, where the procedure can be clarified: After stopping IRQUEUE, make the backup and install it on test system.
We can deploy process monitor tool from sysinternal suite by Microsoft to monitor IR thread performance during regen process.
There do not start irqueue process, as then the irqueue dbdict will have the same state as in production, and after the IR index is promoted to production, and the IRQUEUE process is started again, it will include all IR updates now correctly.
The content of scirexpert file can be copied with RDBMS-utilities, which is also the most efficient method. If you do not ir regen all dbdicts with IR key, you can also restrict the set of record from scirexpert to promote by a query.
The IR performance in general, shared memory requirements for IR index, and for IR regen in specific can only be tuned by having a good maintained stop words file.
The IRQUEUE process is stopped during this procedure: Until new ir index is not promoted to productions, searches will not find anything later than that.
Stopping IRQUEUE process has no other impact to the system other than ir updates will not be processed anymore.
As the ir updates during this time still get queued in irqueue dbdict, after the index is promoted and irqueue process is restarted, this delta will be applied to the ir index on production.
There will be nothing missed. Only be aware, that IRQUEUE process will be very busy for quite a while after being restarted.
Please back up your production database and restore it on test environment and Navigate to the service manager RUN directory and carry out IR Regen against the ir.probsummary and ir.rootcause
Example command to regen ir.probsummary
D:\HP\Service Manager \Server\RUN>sm -util
add: add cls: close cnt: count del: delete dis: display fnd: find get: get nxt: next opn: open pat: patch prv: previous qbe: qbe reg: ir regen rst: reset rmv: remove upd: update vrir: verify IR x: EXIT
Enter your choice: reg
Regen the IR indexes in a file
Enter file name: probsummary dbRegen() for file 'probsummary' completed successfully.
add: add cls: close cnt: count del: delete dis: display fnd: find get: get nxt: next opn: open pat: patch prv: previous qbe: qbe reg: ir regen rst: reset rmv: remove upd: update vrir: verify IR
x: EXIT Example command to regen ir.rootcause
D:\HP\Service Manager\Server\RUN>sm -util
add: add cls: close cnt: count del: delete dis: display fnd: find get: get nxt: next opn: open pat: patch prv: previous qbe: qbe reg: ir regen rst: reset rmv: remove upd: update vrir: verify IR x: EXIT
Enter your choice: reg
Regen the IR indexes in a file
Enter file name: rootcause dbRegen() for file 'rootcause' completed successfully.
add: add cls: close cnt: count del: delete dis: display fnd: find get: get nxt: next opn: open pat: patch prv: previous qbe: qbe reg: ir regen rst: reset rmv: remove upd: update vrir: verify IR
x: EXIT The IR regen does multiple steps:
1) Delete the old index à That’ll empty the scirexpert file for the regenned file 2) Write a new header for the new index à that’ll write (apparently) 13 records into scirexpert for the new index 3) build the new index in private memory à That’ll write nothing to the scirexpert file 4) Apply the index to the database -> that’ll write all the necessary records to scirexpert
Using and maintaining the stop words list is highly recommended to make sure that IR queries and regens are most efficient.
To determine which terms should be added to the stop words file, run the vrir command from sm –util occasionally (may be every two months).
Any term that occurs more than 10,000 times should be added to the stop words files. The IR system does not function reliably with terms that occur >65,000 times.
D:\HP\Service Manager\Server\RUN>sm -util
add: add cls: close cnt: count del: delete dis: display fnd: find get: get nxt: next opn: open pat: patch prv: previous qbe: qbe reg: ir regen rst: reset rmv: remove upd: update vrir: verify IR x: EXIT
Enter your choice: vrir
Validate an IR file
Enter fully qualified name for the IR file: ir.probsummary
IR validation succeeded. See log for details
add: add cls: close cnt: count del: delete dis: display fnd: find get: get nxt: next opn: open pat: patch prv: previous qbe: qbe reg: ir regen rst: reset rmv: remove upd: update vrir: verify IR
x: EXIT
Load the content of scireexpert in production from test server and Run irqueue with ir_disable:0 parameter added in sm.ini .
This is enough no need for IR Regen because after irqueue process stopped, there would be no update/delta in scirexpert of production system.
The ir updates during this time still get queued in irqueue dbdict, after the index is promoted and irqueue process is restarted, this delta will be applied to the ir index on production.
Basically, IR identifies tokens in records and builds a unique form of indexing to retrieve records that match certain search criteria. The more data in the fields defined in the IR Key, the more data that is indexed by IR. With 1,000,000+ records, it is easy to see why an IR regen of this table could take weeks to months to perform an IR regen.
We recommended action is to remove the IR key from all the dbdict for which you don’t want to use text search. Then schedule the IR regen to execute in the far future. Locate and delete the generated schedule record. The IR regen will be done manually at a later date.
Tokens can be removed from being indexed by appending to the *.stp files in the data directory. These are text files that can be edited with tools like notepad or vi. Again, with 1,000,000+ records it is easy to determine that the number of unique tokens in the IR Key fields will be many. Especially if this environment supports multiple languages.
Here is where performing the IR Regen in the test system provides the most benefit.
The next step is to perform the IR Regen on the TEST environment to get an idea of how long the IR Regen will take to complete in the current configuration. This should be done through the sm utilities as if it is done through the client, the session will timeout as it did before.
D) Perform IR regen via the sm utilities. I. Navigate to the ServiceManager RUN Directory. II. Run the command: sm -util III. Enter Choice: opn (open) IV. Enter file name: probsummary V. Verify that the following message is displayed: dbInitRelation() for file 'probsummary' returned 0. VI. If the above message is not displayed, skip to IX VII. Enter Choice: reg (this command is executes the IR regen, and may take daysweeks to complete) IIX. Enter Choice: cls IX. Enter Choice x
We can further improve IR regen performance by identifying what tokens are indexed, and removing the unnecessary tokens by appending them to the *.stp files. In the same TEST environment. Perform the following commands via the utilities.
Output the indexed tokens to the sc.log/sm.log . Service Manager: I. Navigate to the ServiceManager RUN Directory. II. Run the command: sm -util III. Enter Choice: vrir (outputs IR indexed tokens to the sm.log) IV. Enter fully qualified name for the IR file: for example D:\Program Files\HP\Service Manager 9.41\Server\RUN\ir.probsummary V. Enter Choice x (Exit)
The log will contain the name of the token, the frequency used, and other relevant information. Any tokens that the customer does not want indexed need to be added to the *.stp files. After significantly expanding the *.stp files, a regen can be performed again to determine if the regen time meets satisfactory results. If not, consider further expanding the *.stp files by repeating steps D) and E).
An additional action that can be performed is:Part of the delay of performing the IR regen may be related to transaction times with the Oracle database. Because all IR records are stored in the scirexpert table, all tables with IR Keys will contain records in the scirexpert table. As a result the scirexpert table in oracle will become very large. Moving to the old stype of maintaining IR file will separate each file into its own ir.XXXXX physical file. As a result each IR index will be maintained separately, and may prove to be faster than storing the data on Oracle. This can be done via the sm.ini parameter
Service Manager: sm.ini: ir_external_files:1