Automating Bulkload

The scripts attached here will assist in properly automating and scheduling the bulkloading of IdML books. The scripts will run the bulkload against the discovery library with optional flags, check the results for failures, and send an e-mail notification with the results (for Windows).

This has been tested on 7.3 and 7.2.2 on Windows and Linux.

Installation and Usage

Download the following files to your $COLLATION_HOME/custom directory on any of thestorage servers in a streaming environment or the domain server in a domain environment. You may need to create the directory 'custom'. $COLLATION_HOME is by default /opt/IBM/taddm (C:\ibm\taddm\dist on Windows). In this zip file you will find some tools to assist you in automating bulkload of DLA IdML books (XML files).

• abulkload.properties - This is the properties file with values for your environment. Edit and set any flags needed and set the location of your discover library where IdML books are read from.
• abulkload.ps1 - This is the main powershell script that executes the bulkload for Windows.
• abulkload.sh - This is the main shell script that executes the bulkload for Linux. There is a variable named ARCHIVEDELETEAFTER in the top of the file that you can set to the number of days you would like to keep archived books and results. By default this value is set to 0 (do not delete any archived books or results).
• abulkload_wrapper.bat - This script executes abulkload.ps1, captures the output, then executes Notification.ps1 to email the results. This is the script that should be executed via Windows task scheduler. Edit to set the -to and -from e-mail addresses on the last line.
• Notification.ps1 - This is a powershell script to send e-mail notifications via SMTP for Windows. Edit the script to set the default $from variable and $smtpServer variable for your environment.

A recommended best practice is to increase the default processedfiles.cleanup value for the TADDM bulkload. This will prevent the bulkload process from accidentally loading old IdML books that weren't cleaned up from the discovery library. Set com.ibm.cdb.bulk.processedfiles.cleanup=3000 in dist\etc\bulkload.properties. No TADDM restart is required.

Additional Notes for Windows

For these scripts to work on Windows, you will need to install PowerShell if you are running on Windows 2003. Also, make sure that you run 'Set-ExecutionPolicy Unrestricted' for both the 32-bit and 64-bit powershell at the command prompt. This only needs done once.

For Windows, you can simply use the Windows task scheduler to schedule execution of abulkload_wrapper.bat. There is a local security policy setting for Windows 2008 that can prevent the automated script from accessing the other servers on the network. The policy name is "Network access: Do not allow storage of passwords and credentials for network authentication" and is found under "SECPOL.MSC | Security Settings | Local Policies | Security Options". If this is set to enabled then the scripts will be unable to access remote discovery library directories.