Manipulators are groups of motorized axes, typically an XY stage or an XYZ unit. The basic class is a .ManipulatorUnit, which depends on a controller. The controller is a set of axes that can be moved independently (for example the Luigs and Neumann controller, which controls up to 9 axes). Example:
controller = LuigsNeumann_SM10() stage = ManipulatorUnit(controller, [7, 8])
A .ManipulatorUnit can be moved with relative or absolute displacements expressed in µm.
Calibrated units are manipulator units that can be moved in the coordinate system of the camera, called the reference system. A .CalibratedUnit must be associated to a camera, a microscope Z axis, and can be attached to an XY stage. A .CalibratedStage is a special kind that a horizontal XY stage (i.e., to be parallel to the focal plane of the microscope). Examples:
calibrated_stage = CalibratedStage(stage, microscope=microscope, camera=camera) XYZ = CalibratedUnit(unit, calibrated_stage, microscope, camera=camera)
The position in the camera system is given by {\bf M}\cdot{\bf u} + {\bf r}_0 + {\bf r}_S, where {\bf u} is the position of the manipulator axes, {\bf r}_S is the position of the stage in the reference system to which the manipulator is attached, {\bf M} is coordinate change matrix and {\bf r}_0 is an offset.
The basic move is a ~.CalibratedUnit.reference_move. It simply inverts the matrix relation to find the
target position in the coordinate system of the manipulator coordinates.
However, this is not as simple as it sounds. For some manipulators (including Luigs and Neumann),
the resulting displacement is not necessarily a straight line because each axis has a fixed speed independent
of the movement. This can result in a broken trajectory; first a diagonal move then a move in the remaining
directions. As a result, the pipette could collide with the coverslip or other problems.
To avoid this problem, the method has a safe
option. If True, the method first determines whether the
the third axis, which is assumed to be Z, will be moved up or down (see calibration algorithms).
If it is going up, then this axis is moved first; otherwise it is moved last. This simple algorithm
maximizes the minimum altitude of the trajectory, so as to avoid colliding with the coverslip.
This is only done with absolute moves and not relative moves.
The ~.CalibratedUnit.withdraw method moves the first axis to its upper endpoint. This presupposes that the two endpoints have been previously identified.
The ~.CalibratedUnit.focus method moves the microscope Z axis so that the tip is in focus. This does not use an autofocus but rather the calibration system (so the manipulator must be correctly calibrated for this to work).
The ~.CalibratedUnit.safe_move method moves the manipulator to a target point, with a trajectory that aims at minimizing mechanical interaction with the tissue. It also essentially removes the pipette from the field of view during the approach, which could be helpful if tracking the cell.
If the movement is up, a normal movement is done (with the safe
option). If it is down, then the trajectory
is more complex. First, the manipulator is moved horizontally, then along the first axis of the manipulator.
Note that by horizontally, it is meant that the start and end positions are on the same horizontal plane, but
the trajectory does not necessarily remain in that plane for reasons explained above; thus the
safe
option is also used.
If the recalibrate
option is True
and the movement is at least 500 µm, then the program tries to fix
errors in calibration before the target. To this end, the manipulator stops 50 µm before the target,
then focus on the tip, automatically recalibrate (see below), then move the focus back, and finish the movement.
This is not fully tested code. The ~.CalibratedUnit.move_new_pipette_back method moves a new pipette into the field. This assumes that the calibration is right, except for an offset (due e.g. to the length and slightly different geometry of the pipette). The algorithm is as follows:
- Move the pipette 2 mm before target position (in the direction of the first axis), which is the center of the microscope view.
- Take 10 photos at 10 Hz.
- Calculate the mean standard deviation of the images (more or less the contrast).
- Move the pipette down by 100 µm along the axis.
- If the standard deviation of the image differs by at least 20% from the mean calculated previously, stop.
- Otherwise, go to 4; stop at 5 mm.
In practice, if the pipette is cleaned, this method might not be that useful.
The ~.CalibratedUnit.move_and_track is used by calibration algorithms. It moves the pipette along one axis, then focus the microscope on the tip using calculation and then template matching. Optionally, it also moves the stage to center the tip. The final image is focused on the tip, but the tip is not necessarily in the center (depending on the precision of calibration). Finally, it returns the position of the tip on screen and focal plane.
The ~.CalibratedUnit.move_back method is used by calibration algorithms. It moves the microscope, manipulator and stage to a given position (previously stored), in a certain order that is intended to avoid collisions. First, the microscope is moved (normally, up), then the manipulator, then the stage. The pipette is then back at the initial position, which is supposed to be in focus in the center of view. Then the pipette is located and refocused, and the pipette position and focal plane are returned.
Calibration consists in determining the matrix {\bf M} and the offset {\bf r}_0, as well as whether the axes go up or down (in Z) in the positive direction.
This assumes that the manipulator is correctly calibrated, except for an offset. The method ~.CalibratedUnit.recalibrate updates {\bf r}_0 assuming that the tip is in the center of view (red cross), or at the given (x,y) position on screen if provided (right-click on the standard interface).
The stage is assumed to be horizontal, and thus the Z axis of the microscope is not moved. It is assumed that there is an object in focus in the field of view, attached to the stage (pipette, or coverslip). Algorithm:
- Take a photo of the center of the field: this is the template.
- Move the first axis by 40 µm, and locate the template in the image: deduce the first column of {\bf M}.
- Repeat for the second axis.
- Using the first estimate of {\bf M}, move to each of three corners of the image (top left, top right, bottom left), with a safety margin, and locate the template.
- Calculate {\bf M} again based on these three points.
This is the ~.CalibratedUnit.calibrate method, plus a number of methods that it calls. The tip must be in focus at the center of view.
Initial steps
- Calibrate the stage to which it is attached.
- Take photos of the pipette along the Z axis of the microscope, every 1 µm over
distance
stack_depth
(positive and negative).
First estimate
- Move and track the first axis by a distance equal to half the
stack_depth
. As initially the matrix is zero, there is no predictive move of the focus. - Repeat for each axis.
- Calculate the matrix.
- Go back to the initial position.
This first very crude estimate is used to calculate the vertical direction of the axes.
Up directions
This is done in method ~.CalibratedUnit.calculate_up_directions. It takes the matrix and estimates for each axis whether a positive movement makes the pipette go up or down. Then the minimum reachable Z (coverslip) is determined as 300 µm below the current position, unless it has been specified explicitly (floor position).
Calibration
Each axis is calibrated in turn. For each axis:
- Double the movement amplitude.
- Check whether the movement is reachable (which presupposes that ranges have been set).
- Estimate whether the movement will make the pipette move out from the field of view.
- Move the pipette and track, and move the stage to compensate if the pipette is out of field.
- Calculate the relevant column of {\bf M}, based on camera positions before and after the movement.
- Repeat
calibration_moves
times. - Move back to the initial position.
- Calculate the relevant column of {\bf M}, based on camera positions before and after the movement.
Thus, only the last movement (which is the largest one) is actually used to calculate the matrix.
The ~.CalibratedUnit.manual_calibration method takes 4 points chosen by the user, and deduce the matrix from them.
- Locate the pipette over a depth of +-25 µm, using templates and movements of microscope Z.
- Update the offset {\bf r_0} (recalibration).
- With option
center
, move the stage and focus so that the pipette tip is centered.