0readmeAsynchIO.txt

SIM_ASYNCH_IO

Theory of operation.

Features.
   - Optional Use.  Build with or without SIM_ASYNCH_IO defined and 
     simulators will still build and perform correctly when run.
     Additionmally, a simulator built with SIM_ASYNCH_IO defined can
     dynamically disable and reenable asynchronous operation with 
     the scp commands SET NOASYNCH and SET ASYNCH respectively.
   - Consistent Save/Restore state.  The state of a simulator saved 
     on a simulator with (or without) Asynch support can be restored
     on any simulator of the same version with or without Asynch 
     support.
   - Optimal behavior/performance with simulator running with or 
     without CPU idling enabled.
   - Consistent minimum instruction scheduling delays when operating 
     with or without SIM_ASYNCH_IO.  When SIM_ASYNCH_IO is emabled, 
     any operation which would have been scheduled to occurr in 'n' 
     instructions will still occur (from the simulated computer's 
     point of view) at least 'n' instructions after it was initiated.
   
Benefits.
   - Allows a simulator to execute simulated instructions concurrently 
     with I/O operations which may take numerous milliseconds to perform.
   - Allows a simulated device to potentially avoid polling for the 
     arrival of data.  Polling consumes host processor CPU cycles which 
     may better be spent executing simulated instructions or letting 
     other host processes run.  Measurements made of available 
     instruction execution easily demonstrate the benefits of parallel
     instruction and I/O activities.  A VAX simulator with a process 
     running a disk intensive application in one process was able to 
     run (in another process) 11 times the number of Dhrystone operations 
     with Asynch I/O enabled vs not enabled.
   - Allows simulator clock ticks to track wall clock was precisely as 
     possible under varying I/O load and activities.

SimH Libraries which provide Asynch I/O support:
   sim_disk
   sim_tape
   sim_ether
   sim_console
   sim_tmxr

Requirements to use:
The Simulator's instruction loop needs to be modified to include a single
line which checks for asynchronouzly arrived events.  The vax_cpu.c 
module added the following line indicated by >>>:

		/* Main instruction loop */

        for ( ;; ) {

        [...]
>>>	        AIO_CHECK_EVENT;
        	if (sim_interval <= 0) {                /* chk clock queue */
        		temp = sim_process_event ();
        		if (temp)
        			ABORT (temp);
        		SET_IRQL;                           /* update interrupts */
        		}

A global variable (sim_asynch_latency) is used to indicate the "interrupt 
dispatch latency".  This variable is the number of nanoseconds between checks
for completed asynchronous I/O.  The default value is 4000 (4 usec) which
corresponds reasonably with simulated hardware.  This variable controls
the computation of sim_asynch_inst_latency which is the number of simulated
instructions in the sim_asynch_latency interval.  We are trying to avoid
checking for completed asynchronous I/O after every instruction since the 
actual checking every instruction can slow down execution.  Periodic checks
provide a balance which allows response similar to real hardware while also 
providing minimal impact on actual instruction execution.  Meanwhile, if 
maximal response is desired, then the value of sim_asynch_latency can be 
set sufficiently low to assure that sim_asynch_inst_latency computes to 1.
The sim_asynch_inst_latency is dynamically updated once per second in the 
sim_rtcn_calb routine where clock to instruction execution is dynamically 
determined.  A simulator would usually add register definitions
to enable viewing and setting of these variables via scp:

#if defined (SIM_ASYNCH_IO)
    { DRDATA (LATENCY, sim_asynch_latency, 32), PV_LEFT },
    { DRDATA (INST_LATENCY, sim_asynch_inst_latency, 32), PV_LEFT },
#endif

Programming Disk and Tape devices to leverage Asynch I/O

Asynch disk and tape I/O is provided through a callback model.  The callback
is invoked when the desired I/O operation has completed.

Naming conventions:
All of the routines implemented in sim_disk and sim_tape have been kept
in place.  All routines which perform I/O have a variant routine available
with a "_a" appended to the the routine name with the addition of a single
parameter which indicates the asynch completion callback routine.  For 
example there now exists the routines:
  t_stat sim_tape_rdrecf (UNIT *uptr, uint8 *buf, t_mtrlnt *bc, t_mtrlnt max);
  t_stat sim_tape_rdrecf_a (UNIT *uptr, uint8 *buf, t_mtrlnt *bc, t_mtrlnt max, TAPE_PCALLBACK callback);

The Purpose of the callback function is to record the I/O completion status
and then to schedule the activation of the unit.  

Considerations:
Avoiding multiple concurrent users of the unit structure.  While asynch
I/O is pending on a Unit, the unit should not otherwise be on the event 
queue.  The I/O completion will cause the Unit to be scheduled to run
immediately to actually dispatch control flow to the callback routine.
The callback routine is always called in the same thread which is 
executing instructions.  Since all simulator device data structures are 
only referenced from this thread there are no host multi-processor cache 
coherency issues to be concerned about.

Arguments to the callback routine:
UNIT *, and IO Status
Requirements of the Callback routine.
The callback routine must save the I/O completion status in a place
which the next invocation of the unit service routine will reference 
and act on it.  This allows device code to return error conditions
back to scp in a consistent way without regard to how the callback 
routine (and the actual I/O) may have been executed.  When the callback
routine is called, it will already be on the simulator event queue with
an event time which was specified when the unit was attached or via a 
call to sim_disk_set_async.  If no value has been specified then it
will have been scheduled with a delay time of 0.  If a different event 
firing time is desired, then the callback completion routine should 
call sim_activate_abs to schedule the event at the appropriate time.

Required change in device coding.
Devices which wish to leverage the benefits of asynch I/O must rearrange
the code which implements the unit service routine.  This rearrangement
usually entails breaking the activities into two phases.  The first phase
(I'll call the top half) involves performing whatever is needed to 
initiate a call to perform an I/O operation with a callback argument.  
Control is then immediately returned to the scp event dispatcher.
The callback routine needs to be coded to stash away the io completion 
status and some indicator that an I/O has completed.
The top/bottom half separation of the unit service routine would be
coded to examine the I/O completion indicator and invoke the bottom half
code upon completion.  The bottom half code should clear the I/O 
completion indicator and then perform any activities which normally 
need to occur after the I/O completes.  Care should be taken while
performing these top/bottom half activities to return to the scp event
dispatcher with either SCPE_OK or an appropriate error code when needed.  
The need to return error indications to the scp event dispatcher is why 
the bottom half activities can't simply be performed in the 
callback routine (the callback routine does not return a status). 
Care should also be taken to realize that local variables in the 
unit service routine will not directly survive between the separate 
top and bottom half calls to the unit service routine.  If any such
information must be referenced in both the top and bottom half code paths
then it must either be recomputed prior to the top/bottom half check
or not stored in local variables of the unit service routine.

Sample Asynch I/O device implementations.
The pdp11_rq.c module has been refactored to leverage the asynch I/O
features of the sim_disk library.  The impact to this code to adopt the
asynch I/O paradigm was quite minimal.
The pdp11_rp.c module has also been refactored to leverage the asynch I/O
features of the sim_disk library.  The impact to this code to adopt the
asynch I/O paradigm was also quite minimal.  After conversion a latent
bug in the VAX Massbus adapter implementation was illuminated due to the
more realistic delays to perform I/O operations.
The pdp11_tq.c module has been refactored to leverage the asynch I/O
features of the sim_tape library.  The impact to this code to adopt the
asynch I/O paradigm was very significant. This was due to the two facts:
1) there are many different operations which can be requested of tape 
devices and 2) some of the tmscp operations required many separate 
operations on the physical device layer to perform a single tmscp request.
This issue was addressed by adding additional routines to the physical 
device layer (in sim_tape.c) which combined these multiple operations.
This approach will dovetail well with a potential future addition of 
operations on physical tapes as yet another supported tape format.

Programming Console and Multiplexer devices to leverage Asynch I/O to 
minimize 'unproductive' polling.

There are two goals for asynchronous Multiplexer I/O: 1) Minimize polling
to only happen when data is available, not arbitrarily on every clock tick, 
and 2) to have polling actually happen as soon as data may be available.  
In most cases no effort is required to add Asynch I/O support to a 
multiplexer device emulation.  If a device emulation takes the normal 
model of polling for arriving data on every simulated clock tick, then if
Asynch I/O is enabled, the device will operate asynchronously and behave 
well.  There is one restriction in this model.  Specifically,  the device 
emulation logic can't expect that there will be a particular number (clock 
tick rate maybe) of invocations of a unit service routine to perform polls 
in any interval of time (this is what we're trying to change, right?).  
Therefore presumptions about measuring time by counting polls is not 
valid.  If a device needs to manage time related activities, then the 
device should create a separate unit which is dedicated to the timing 
activities and which explicitly schedules a different unit service routine 
for those activities as needed.  Such scheduled polling should only be 
enabled when actual timing is required. 

A device which is unprepared to operate asynchronously can specifically
disable multiplexer Asynch I/O for that device by explicitly defining 
NO_ASYNCH_MUX at compile time.  This can be defined at the top of a 
particular device emulation which isn't capable of asynch operation, or 
it can be defined globally on the compile command line for the simulator.
Alternatively, if a specific Multiplexer device doesn't function correctly 
under the multiplexer asynchronous environment and it will never be 
revised to operate correctly, it may statically set the TMUF_NOASYNCH bit 
in its unit flags field.

Some devices will need a small amount of extra coding to leverage the 
Multiplexer Asynch I/O capabilties.  Devices which require extra coding
have one or more of the following characteristics:
- they poll for input data on a different unit (or units) than the unit 
  which was provided when tmxr_attach was called.
- they poll for connections on a different unit than the unit which was
  provided when tmxr_attach was called.

The extra coding required for proper operation is to call 
tmxr_set_line_unit() to associate the appropriate input polling unit to 
the respective multiplexer line (ONLY if input polling is done by a unit
different than the unit specified when the MUX was attached). If output 
polling is done on a different unit, then tmxr_set_line_output_unit()
should be called to describe that fact.

Console I/O can operate asynchronously if the simulator notifies the 
tmxr/console subsystem which device unit is used by the simulator to poll 
for console input and output units.  This is done by including sim_tmxr.h 
in the source module which contains the console input device definition 
and calling tmxr_set_console_units().  tmxr_set_console_units would usually 
be called in a device reset routine.

sim_tmxr consumers:
  - Altair Z80 SIO   devices = 1, units = 1,      lines = 4,  flagbits = 8, Untested Asynch
  - HP2100 BACI      devices = 1, units = 1,      lines = 1,  flagbits = 3, Untested Asynch
  - HP2100 MPX       devices = 1, units = 10,     lines = 8,  flagbits = 2, Untested Asynch
  - HP2100 MUX       devices = 3, units = 1/16/1, lines = 16, flagbits = 4, Untested Asynch
  - I7094 COM        devices = 2, units = 4/33,   lines = 33, flagbits = 4, Untested Asynch
  - Interdata PAS    devices = 2, units = 1/32,   lines = 32, flagbits = 3, Untested Asynch
  - Nova QTY         devices = 1, units = 1,      lines = 64, flagbits = 1, Untested Asynch
  - Nova TT1         devices = 2, units = 1/1,    lines = 1,  flagbits = 1, Untested Asynch
  - PDP-1 DCS        devices = 2, units = 1/32,   lines = 32, flagbits = 0, Untested Asynch
  - PDP-8 TTX        devices = 2, units = 1/4,    lines = 4,  flagbits = 0, Untested Asynch
  - PDP-11 DC        devices = 2, units = 1/16,   lines = 16, flagbits = 5, Untested Asynch
  - PDP-11 DL        devices = 2, units = 1/16,   lines = 16, flagbits = 3, Untested Asynch
  - PDP-11 DZ        devices = 1, units = 1/1,    lines = 32, flagbits = 0, Good Asynch
  - PDP-11 VH        devices = 1, units = 4,      lines = 32, flagbits = 4, Good Asynch
  - PDP-18b TT1      devices = 2, units = 1/16,   lines = 16, flagbits = 0, Untested Asynch
  - SDS MUX          devices = 2, units = 1/32,   lines = 32, flagbits = 0, Untested Asynch
  - sim_console                                                             Good Asynch
  
Program Clock Devices to leverage Asynsh I/O

simh's concept of time is calibrated by counting the number of 
instructions which the simulator can execute in a given amount of wall 
clock time.  Once this is determined, the appropriate value is continually 
recalibrated and used throughout a simulator to schedule device time 
related delays as needed.  Historically, this was fine until modern 
processors started having dynamically variable processor clock rates.
On such host systems, the simulator's concept of time passing can vary 
drastically.  This dynamic adjustment of the host system's execution rate
may cause dramatic drifting of the simulated operating system's concept 
of time.  Once all devices are disconnected from the calibrated clock's 
instruction count, the only concern for time in the simulated system is 
that it's clock tick be as accurate as possible.  This has worked well 
in the past, however each simulator was burdened with providing code 
which facilitated managing the concept of the relationship between the 
number of instructions executed and the passage of wall clock time.  
To accomodate the needs of activities or events which should be measured 
against wall clock time (vs specific number of instructions executed), 
the simulator framework has been extended to specifically provide event 
scheduling based on elapsed wall time. A new API can be used by devices 
to schedule unit event delivery after the passage of a specific amount 
of wall clock time.  The api sim_activate_after() provides this 
capability.  This capability is not limited to being available ONLY when 
compiling with SIM_SYNCH_IO defined.  When SIM_ASYNCH_IO is defined, this 
facility is implemented by a thread which drives the delivery of these 
events from the host system's clock ticks (interpolated as needed to 
accomodate hosts with relatively large clock ticks).  When SIM_ASYNCH_IO 
is not defined, this facility is implemented using the traditional simh 
calibrated clock approach.  This new approach has been measured to provide
clocks which drift far less than the drift realized in prior simh versions.  
Using the released simh v3.9-0 vax simulator with idling enabled, the clock 
drifted some 4 minutes in 35 minutes time (approximately 10%).  The same OS 
disk also running with idling enabled booted for 4 hours had less that 5 
seconds of clock drift (approximately 0.03%).

Co-Scheduling Clock and Multiplexer (or other devices)

Many simulator devices have needs to periodically executed with timing on the
order of the simulated system's clock ticks.  There are numerous reasons for
this type of execution.  Meanwhile, many of these events aren't particular 
about exactly when they execute as long as they execute frequently enough.
Frequently executing events has the potential to interfere with a simulator's
attempts to idle when the simulated system isn't actually doing useful work.

Interactions with attempts to 'co-schedule' multiplexer polling with clock
ticks can cause strange simulator behaviors.  These strange behaviors only 
happen under a combination of conditions:
  1) a multiplexer device is defined in the simulator configuration,
  2) the multiplexor device is NOT attached, and thus is not being managed by
     the asynchronous multiplexer support
  3) the multiplexer device schedules polling (co-scheduled) when not 
     attached (such polling will never produce any input, so this is probably
     a bug).
In prior simh versions support for clock co-scheduling was implmented 
separately by each simulator, and usually was expressed by code of the form:
    sim_activate (uptr, clk_cosched (tmxr_poll));
As a part of asynchronous timer support, the simulator framework has been 
extended to generically provide clock co-scheduling support.  The use of this
new capability requires an initial call (usually in the clock device reset 
routing) of the form: 
    sim_register_clock_unit (&clk_unit);
Once the clock unit has been registered, co-scheduling is achieved by replacing
the earlier sim_activate with the following:
    sim_clock_coschedule (&dz_unit, tmxr_poll);

Run time requirements to use SIM_ASYNCH_IO.
The Posix threads API (pthreads) is required for asynchronous execution.
Most *nix platforms have these APIs available and on these platforms
simh is typically built with these available since on these platforms, 
pthreads is required for simh networking support.  Windows can also 
utilize the pthreads APIs if the compile and run time support for the
win32Pthreads package has been installed on the build system.