Skip to content

Dynamic Playback set up and usage page

Jump to bottom
pgale61 edited this page Aug 10, 2022 · 32 revisions

Welcome to the Dynamic_Playback wiki!

Developed by Actian Services

Contents

  1. Playback status
  2. What Playback does
  3. Contact
  4. Download Binaries
  5. Features
  6. Known Issues / gotchas
  7. ToDo list (future versions)
  8. How to install from a binary package
  9. How to install from source
  10. Getting started
  11. Tested Platforms

Welcome to the Playback project

Playback status

Playback has been tested very thoroughly on Ingres 10.2, RHEL 6 just recently. We found an Ingres bug that has existed forever but only showed itself under the load we were able to place on it with our latest servers and a dataset with heavy throughput and contention between sessions.

What Playback does

The tool is designed to record the workload of an Ingres system as it happens - in a live, production environment - and then allow that workload to be played back at a later time on another system, with the goal of being able to test your system with a good representation of what your real-world production workload looks like. It takes the output created by the DBMS Server Query Tracing (SC930) and plays it back into an Ingres Installation. It does that meeting the following criteria:

  • The playback can be simulated exactly as it happened in real or can be accelerated to be faster than real-time.
  • The statements can be forced to play back in the same order as they were originally recorded.

It is not guaranteed that the queries will achieve precisely the same results as they did in the original production run, because there may be subtle variations in record locking behaviour when playing back due to hardware or environmental differences. It is recommended not to force the exact order of playback.

Note: Dynamic Playback is an OpenSource community product and as such is not supported as part of your Ingres subscription. If you need support for this product then Ingres can provide expert consultants to assist with the product. Please see the 'Contact' section for more information.

Contact

Questions about Playback? Drop us a mail: playback@actian.com Note: Dynamic Playback is an OpenSource community product and as such is not supported as part of your Ingres subscription. If you need support for this product then Ingres can provide expert consultants to assist with the product. Please contact me for more information. You can also ask any questions in the Forum

Download Binaries

Download the latest binaries from Github.

If you need binaries for other platforms, please let me know. Status The following platforms have been tested:

  • Preprocessing
  • Linux: Mono 1.9 or higher
  • Windows: .NET Framework 2.0
  • Playback
  • Windows
  • Linux
  • Linux x86-64
  • HPUX
  • AIX
  • Solaris More Platforms might be looked at in later versions. The development language is C with Embedded SQL and Mono compatible C# for the Preprocessing.

Features

The features of the current stable version are:

  • Parsing the output from SC930 and play it back using a multi threaded approach
  • Play back in real time and accelerated time
  • Play back as fast as possible with no delays between the statements
  • Replicate standard SQL, DML and DDL
  • Replicating Repeated SQL
  • Replicating Cursor handling
  • Printing Ingres error output to the console
  • Print number of affected rows to the console
  • Capture Load Statistics
  • Synchronising all statements to ensure they run in the same order as they originally have. Feature is optional and needs to be switched on.
  • It will process Vector and VectorH statements in the same way.

Known Issues / gotchas

  • Cannot handle blobs without existing datahandler(s) linked in. If multiple datahandlers then may have to hack the SQL files to match. If no data or different data flowing through datahandler then cannot guarantee results.
  • Cannot handle copy.in / copy.out of files unless both the files and the directories paths exist as they would in production.
  • Cannot run 32bit Ingreslib.* with 64 bit env and vice versa.
  • Cannot process external statements like copy.in or vwload.

ToDo list (future versions)

  • Create a feature-complete Unit Test
  • Playback needs to simulate all the communication channels to the DBMS that were originally used (Local, Ingres/NET, DAS, ODBC...). This probably calls for multiple synced up executables in different programming languages.
  • Use the Socket Communication feature for a Stats/Progress GUI for Playback?

How to install from a binary package

There is no installation required. Just download the binaries.

Note: This may not be readily available and to be safe you are probably better off compiling on your own server, esp. if using an nux other than Linux, as we don’t have all the variations that customers will have. Esp. if you are running older hardware.

The run time requirements are:

Preprocessing

  1. Linux: Mono Framework 1.9 or higher
  2. Windows: .NET Framework 2.0

Playback

  1. An Ingres installation set to 32/64 bit running, charset, date format, etc. also need to be set as per the query recording install.

How to install from source

Step 1: Get the code

Download a source package here. This code has been thoroughly tested and should be stable.

After downloading you will have a directory called Dynamic_Playback with multiple versions of the code.

[Dynamic_Playback] % ls -l
total 29064
-rw-r--r--  1 petergale  staff    18046 10 Aug 09:47 LICENSE
drwxr-xr-x  3 petergale  staff       96 10 Aug 09:47 Linux
-rw-r--r--  1 petergale  staff      132 10 Aug 09:47 README.md
-rw-r--r--  1 petergale  staff  1531887 10 Aug 09:47 playback-2.5.1.tgz
-rw-r--r--  1 petergale  staff  1658434 10 Aug 09:47 playback-2.5.3.tgz
-rw-r--r--  1 petergale  staff  1700673 10 Aug 09:47 playback-2.5.4.tgz
-rw-r--r--  1 petergale  staff  3024551 10 Aug 09:47 playback-2.5.5.tgz
-rw-r--r--  1 petergale  staff  3019593 10 Aug 09:47 playback-2.5.6.tgz
-rw-r--r--  1 petergale  staff  2254054 10 Aug 09:47 playback-src-2.4.8.zipx
-rw-r--r--  1 petergale  staff  1654650 10 Aug 09:47 playback-src-2.4.9.tgz

Unzip the latest version, e.g. tar xvf playback-2.5.6.tgz, then change to the top level directory before compiling

cd playback-2.5.6/main

NOTE: To compile the development version, you will need GNU autoconf to create the configure script:

 cd Playback
 autoconf

Step 2: Compile and install

If you want to compile the sources you need to have the following prerequisites:

Preprocessing

  • Mono 2.0 or .NET Framework 2.0

Playback

  • SUS2 Compliant C compiler and libs
  • GNU make or compatible (for Unix)
  • MS Visual C++ 2005 (for Windows)
  • Ingres ESQL/C Preprocessor

To build the package on Linux/Unix

  • Use an account that has an Ingres Environment sourced
  • You can check by typing: ingstatus
  • To build just go to the project's root directory and type make
  • If you only want to create a specific module, type: make playback or make preprocess

There are no warnings when compiling on Linux using gcc (tested on Ubuntu 9.04 with gcc 4.3.3). Other compilers might issue warnings that will probably be harmless.

To build on Windows:

  • Playback
    • Use the provided .sln file to build using Visual C++ Express 2008 or higher
  • Preprocessing
    • Is supported and should build using Visual C# Express 2008 or higher

Add to PATH

After building you need to extend the PATH to pick up Preprocess.exe and playback. Do not be tempted to copy/move the executables to somewhere else.

For Preprocess add the absolute path to Dynamic_Playback-master/playback-2.5.6/main/Preprocess/Preprocess/bin/Debug to the PATH.

For playback add he absolute path to Dynamic_Playback-master/Linux/bin to the PATH.

Getting started

Step 1: Getting session files from SC930

See DBMS Server Query Tracing for information about how to capture playback trace files.

Step 2: Preprocess

Go to the directory that contains your session files and run the Preprocessor:

Preprocess.exe

This will generate one *.psf file per session. Preprocessing can be used with the following command line switches:

  • --path=/path/to/sessfiles A switch indicating the path to the session files. It will also be the output path of the *.psf files
  • --dryrun If you specify this switch, no .psf files will be created. The tool will output additional information on the console.
  • --filterdb=123456789 This will parse only sessions from the database with the ID 123456789. All other sessions will be ignored. If you want to find out, what IDs your databases have, go to your iidbdb and issue this statement: select db_id, name from iidatabase
  • --dbid:123456789=myDatabase This is needed to put database names into the parsed session files. You can add as many --dbid switches as you need NOTE: Setting aliases for DBIDs is important if you want to play back against multiple databases. If you don't use this switch you will only be able to play back against one database!

Step 3: Playing it back

Recover database to starting point. Change to the directory that contains the *.psf files. Run the command playback . e.g.

./playback -v -S1024 -l32768 -dmytestdb -umyuser -s50 -t50 > ./playback.log 2>./playback.err

Here is a list of the command line options:

  • -o Runmode.
  • -o0 runs the queries as fast as possible (no timings, no delays),
  • -o1 runs them in real time simulation. (default)
  • -v Verbose. Displays extra output
  • -t Step count. This is the number of simulated milliseconds, a simulated tick would jump ahead
  • -s Step delay. This is the number of milliseconds that a simulated tick would take in real time.
  • For example, -t10 -s100 would play back 10ms of the trace in 100ms of real time. The default is -t1000 -s1000.
  • -d Name of the database to connect to
  • -u User name to use for the connection. Default is ingres
  • Note: The database and user name here are only used when the session file contains no db or user information!
  • -z Switch on synchronising. This forces the statements to be executed in the original order. The feature is disabled by default.
  • -m Mock mode. This is useful if you want to test the tool and have no session files yet. It will run two concurrent sessions issuing very simple catalog queries. It won't break anything.
  • -R Ignore repeated queries. Repeated queries are treated as ordinary non-repeated ones.
  • -f Turn statistics writing off. The file playbackrun.sh will not be created
  • -S default Stack in KB’s
  • -l Max size of LOB retrieved from DB. A too small value could result in memory issues.

Also note you may want to set stack and file limits before starting. e.g.

ulimit -n 131072

ulimit -s 1048576

Tested Platforms

Preprocessor

Preprocessor Version	Platform / OS	MONO/VS.Net 	Ingres Version	Status  
2.1.12			Linux		2.4		9.2		Success  
2.1.10			Ubuntu		2.0.1		9.2		Success  
2.3.2			Solaris 10	2.8		9.2		Success  
2.3.2			RHEL 4		2.3		9.2		Issues  
2.3.2			Windows	2003 	.net 2.0	9.2		Success  
2.3.2			RHEL 5				2.6		Success  
2.4.0			RHEL 5				2.6		Success  
2.4.2			RHEL 5				9.2		Success  
2.4.3			RHEL 5				9.2		Success  
2.4.3			RHEL 5				9.3		Success  
2.4.6			RHEL 6				10.0		Success  
2.4.7			RHEL 6				10.2		Success

Playback

Playback	Ingres Version			  	Platform / OS		Compiler		Status  
Version  
2.1.12	2.6 (recording / playback) 9.2 (playback)	Tru64 / Solaris 10	Compact 2.29 / Sun Studio 5.29	Success  
2.1.10	2.6/0604 (recording / playback) 9.2 (playback)	HP/UX 11.11/11.23	gcc 4.1.1			Success  
2.3.2	2.6/0305 (recording) 		9.2 (playback)	Solaris 10		gcc 3.4.6			Issues  
2.3.2	2.6 and 9.2					Tru64 5.1B and RHEL 4	gcc 4.1.2			Issues  
2.3.2	2.6 and 9.2					AIX i64.hpu r64.lnx	??				Success  
2.4.0	9.2						RHEL 4			gcc 4.1.2			Success  
2.4.2	9.2						RHEL 5			gcc 4.1.2			Success  
2.4.3	9.2						RHEL 5			gcc 4.1.2			Success  
2.4.3	9.3						RHEL 5			gcc 4.1.2			Success  
2.4.6	10.0						RHEL 6			gcc 4.1.2			Success  
2.4.8	10.2						RHEL 6			gcc 4.4.7			Success
Clone this wiki locally