🐛 ANTs doesn't respect memory constraints #1404
Comments
|
I don't know enough to know how much memory is reasonable, but I hear
|
|
There are some clear memory spikes for ANTs when it runs on Brainlife (this is one subject with
|
|
I'm not sure I quite understand [{
"name": "resting_preproc_sub-A00013809_ses-DS2.nuisance_regressor_0_0.aCompCor_cosine_filter",
"time": 1605723308.305644,
"rss_GiB": 36.45994949316406,
"cpus": 103.0,
"vms_GiB": 40.09559631347656,
"interface": "Function",
"params": "_scan_rest_acq-645__selector_WM-2mm-M_CSF-2mm-M_tC-5PCT2-PC5_aC-CSF+WM-2mm-PC5_G-M_M-SDB_P-2_BP-B0.01-T0.1",
"mapnode": 0
}, {
"name": "resting_preproc_sub-A00013809_ses-DS2.nuisance_regressor_0_0.aCompCor_cosine_filter",
"time": 1605723285.300852,
"rss_GiB": 28.016590118164064,
"cpus": 832.8,
"vms_GiB": 40.09559631347656,
"interface": "Function",
"params": "_scan_rest_acq-645__selector_WM-2mm-M_CSF-2mm-M_tC-5PCT2-PC5_aC-CSF+WM-2mm-PC5_G-M_M-SDB_P-2_BP-B0.01-T0.1",
"mapnode": 0
}, {
"name": "resting_preproc_sub-A00013809_ses-DS2.nuisance_regressor_0_0.aCompCor_cosine_filter",
"time": 1605723284.15847,
"rss_GiB": 27.693824767578125,
"cpus": 2742.9,
"vms_GiB": 31.022430419921875,
"interface": "Function",
"params": "_scan_rest_acq-645__selector_WM-2mm-M_CSF-2mm-M_tC-5PCT2-PC5_aC-CSF+WM-2mm-PC5_G-M_M-SDB_P-2_BP-B0.01-T0.1",
"mapnode": 0
}] |
|
Another clue, interacting with the graphs on brainlife, I see it's actually
|
|
Those brief spikes are huge!!! Do you also mean that it probably is the run.py, not the ANTs? |
|
Not sure. I think it's probably how C-PAC is allocating memory for ANTs |
|
After @ccraddock explained to me that
I started to set
There are still some spikes, but they're much less dramatic. I'm iterating now, setting estimates based on logged memory usage after fresh runs after setting more estimates. |
|
This is looking good. I think that there is an issue with the number of threads being estimated by the callback, or the gantt chart creation script is pulling in the wrong numbers. Some of the nodes are reporting using 210 threads! As for your earlier comment on run.py, I think that since this is the parent process, it 'owns' all of the memory used by the child threads. So the amount of memory attributed to it should be the cumulative amount of memory used by all of the nodes that are currently executing. |
|
I thought maybe I see the profile uses I'll try updating the callback to get the actual number of threads, run a few C-PAC runs and see how it looks. |
|
After a little tinkering, I think estimating the number of threads (by dividing by
I'm putting a PR in to Nipype to restore the Gantt chart creation capabilities, and I'll note the thread-logging ambiguity there. |
|
Awesome @shnizzedy !! |
|
This looks like it's working for memory.
I'm going through now and adding
and adding to the developer docs as I go |
Describe the bug
This is particularly a problem on clusters that have hard memory limits.
Here's an example
command.txtthat uses too much memory:(same thing, wrapped for ease of reading):
To Reproduce
Steps to reproduce the behavior:
--preconfig fmriprep-optionsn_cpusandmem_gbmem_gbexceeded by ANTsExpected behavior
ANTs uses no more than
mem_gbmemory at a time.Versions
Additional context
These issues are almost certainly a result of this issue:
montage_a&montage_s#1338Possibly related: #1054, nipy/nipype#2776
— My registration fails with an error: Memory errors. ANTs wiki
There seems to be no specific memory limitations in the
antsRegistrationcommand apart from--float:$ antsRegistration --help COMMAND: antsRegistration This program is a user-level registration application meant to utilize classes in ITK v4.0 and later. The user can specify any number of "stages" where a stage consists of a transform; an image metric; and iterations, shrink factors, and smoothing sigmas for each level. Note that explicitly setting the dimensionality, metric, transform, output, convergence, shrink-factors, and smoothing-sigmas parameters is mandatory. OPTIONS: --version Get Version Information. -d, --dimensionality 2/3/4 This option forces the image to be treated as a specified-dimensional image. If not specified, we try to infer the dimensionality from the input image. -o, --output outputTransformPrefix [outputTransformPrefix,<outputWarpedImage>,<outputInverseWarpedImage>] Specify the output transform prefix (output format is .nii.gz ). Optionally, one can choose to warp the moving image to the fixed space and, if the inverse transform exists, one can also output the warped fixed image. Note that only the images specified in the first metric call are warped. Use antsApplyTransforms to warp other images using the resultant transform(s). -j, --save-state saveSateAsTransform Specify the output file for the current state of the registration. The state file is written to an hdf5 composite file. It is specially usefull if we want to save the current state of a SyN registration to the disk, so we can load and restore that later to continue the next registration process directly started from the last saved state. The output file of this flag is the same as the write-composite-transform, unless the last transform is a SyN transform. In that case, the inverse displacement field of the SyN transform is also added to the output composite transform. Again notice that this file cannot be treated as a transform, and restore-state option must be used to load the written file by this flag. -k, --restore-state restoreStateAsATransform Specify the initial state of the registration which get immediately used to directly initialize the registration process. The flag is mutually exclusive with other intialization flags.If this flag is used, none of the initial-moving-transform and initial-fixed-transform cannot be used. -a, --write-composite-transform 1/(0) Boolean specifying whether or not the composite transform (and its inverse, if it exists) should be written to an hdf5 composite file. This is false by default so that only the transform for each stage is written to file. <VALUES>: 0 -p, --print-similarity-measure-interval <unsignedIntegerValue> Prints out the CC similarity metric measure between the full-size input fixed and the transformed moving images at each iteration a value of 0 (the default) indicates that the full scale computation should not take placeany value greater than 0 represents the interval of full scale metric computation. <VALUES>: 0 --write-interval-volumes <unsignedIntegerValue> Writes out the output volume at each iteration. It helps to present the registration process as a short movie a value of 0 (the default) indicates that this option should not take placeany value greater than 0 represents the interval between the iterations which outputs are written to the disk. <VALUES>: 0 -z, --collapse-output-transforms (1)/0 Collapse output transforms. Specifically, enabling this option combines all adjacent transforms wherepossible. All adjacent linear transforms are written to disk in the forman itk affine transform (called xxxGenericAffine.mat). Similarly, all adjacent displacement field transforms are combined when written to disk (e.g. xxxWarp.nii.gz and xxxInverseWarp.nii.gz (if available)).Also, an output composite transform including the collapsed transforms is written to the disk (called outputCollapsed(Inverse)Composite). <VALUES>: 1 -i, --initialize-transforms-per-stage (1)/0 Initialize linear transforms from the previous stage. By enabling this option, the current linear stage transform is directly intialized from the previous stage's linear transform; this allows multiple linear stages to be run where each stage directly updates the estimated linear transform from the previous stage. (e.g. Translation -> Rigid -> Affine). <VALUES>: 0 -n, --interpolation Linear NearestNeighbor MultiLabel[<sigma=imageSpacing>,<alpha=4.0>] Gaussian[<sigma=imageSpacing>,<alpha=1.0>] BSpline[<order=3>] CosineWindowedSinc WelchWindowedSinc HammingWindowedSinc LanczosWindowedSinc GenericLabel[<interpolator=Linear>] Several interpolation options are available in ITK. These have all been made available. Currently the interpolator choice is only used to warp (and possibly inverse warp) the final output image(s). -g, --restrict-deformation PxQxR This option allows the user to restrict the optimization of the displacement field, translation, rigid or affine transform on a per-component basis. For example, if one wants to limit the deformation or rotation of 3-D volume to the first two dimensions, this is possible by specifying a weight vector of '1x1x0' for a deformation field or '1x1x0x1x1x0' for a rigid transformation. Low-dimensional restriction only works if there are no preceding transformations.All stages up to and including the desired stage must have this option specified,even if they should not be restricted (in which case specify 1x1x1...) -q, --initial-fixed-transform initialTransform [initialTransform,<useInverse>] [fixedImage,movingImage,initializationFeature] Specify the initial fixed transform(s) which get immediately incorporated into the composite transform. The order of the transforms is stack-esque in that the last transform specified on the command line is the first to be applied. In addition to initialization with ITK transforms, the user can perform an initial translation alignment by specifying the fixed and moving images and selecting an initialization feature. These features include using the geometric center of the images (=0), the image intensities (=1), or the origin of the images (=2). -r, --initial-moving-transform initialTransform [initialTransform,<useInverse>] [fixedImage,movingImage,initializationFeature] Specify the initial moving transform(s) which get immediately incorporated into the composite transform. The order of the transforms is stack-esque in that the last transform specified on the command line is the first to be applied. In addition to initialization with ITK transforms, the user can perform an initial translation alignment by specifying the fixed and moving images and selecting an initialization feature. These features include using the geometric center of the images (=0), the image intensities (=1), or the origin of the images (=2). -m, --metric CC[fixedImage,movingImage,metricWeight,radius,<samplingStrategy={None,Regular,Random}>,<samplingPercentage=[0,1]>] MI[fixedImage,movingImage,metricWeight,numberOfBins,<samplingStrategy={None,Regular,Random}>,<samplingPercentage=[0,1]>] Mattes[fixedImage,movingImage,metricWeight,numberOfBins,<samplingStrategy={None,Regular,Random}>,<samplingPercentage=[0,1]>] MeanSquares[fixedImage,movingImage,metricWeight,radius=NA,<samplingStrategy={None,Regular,Random}>,<samplingPercentage=[0,1]>] Demons[fixedImage,movingImage,metricWeight,radius=NA,<samplingStrategy={None,Regular,Random}>,<samplingPercentage=[0,1]>] GC[fixedImage,movingImage,metricWeight,radius=NA,<samplingStrategy={None,Regular,Random}>,<samplingPercentage=[0,1]>] ICP[fixedPointSet,movingPointSet,metricWeight,<samplingPercentage=[0,1]>,<boundaryPointsOnly=0>] PSE[fixedPointSet,movingPointSet,metricWeight,<samplingPercentage=[0,1]>,<boundaryPointsOnly=0>,<pointSetSigma=1>,<kNeighborhood=50>] JHCT[fixedPointSet,movingPointSet,metricWeight,<samplingPercentage=[0,1]>,<boundaryPointsOnly=0>,<pointSetSigma=1>,<kNeighborhood=50>,<alpha=1.1>,<useAnisotropicCovariances=1>] IGDM[fixedImage,movingImage,metricWeight,fixedMask,movingMask,<neighborhoodRadius=0x0>,<intensitySigma=0>,<distanceSigma=0>,<kNeighborhood=1>,<gradientSigma=1>] These image metrics are available--- CC: ANTS neighborhood cross correlation, MI: Mutual information, Demons: (Thirion), MeanSquares, and GC: Global Correlation. The "metricWeight" variable is used to modulate the per stage weighting of the metrics. The metrics can also employ a sampling strategy defined by a sampling percentage. The sampling strategy defaults to 'None' (aka a dense sampling of one sample per voxel), otherwise it defines a point set over which to optimize the metric. The point set can be on a regular lattice or a random lattice of points slightly perturbed to minimize aliasing artifacts. samplingPercentage defines the fraction of points to select from the domain. In addition, three point set metrics are available: Euclidean (ICP), Point-set expectation (PSE), and Jensen-Havrda-Charvet-Tsallis (JHCT). -t, --transform Rigid[gradientStep] Affine[gradientStep] CompositeAffine[gradientStep] Similarity[gradientStep] Translation[gradientStep] BSpline[gradientStep,meshSizeAtBaseLevel] GaussianDisplacementField[gradientStep,updateFieldVarianceInVoxelSpace,totalFieldVarianceInVoxelSpace] BSplineDisplacementField[gradientStep,updateFieldMeshSizeAtBaseLevel,<totalFieldMeshSizeAtBaseLevel=0>,<splineOrder=3>] TimeVaryingVelocityField[gradientStep,numberOfTimeIndices,updateFieldVarianceInVoxelSpace,updateFieldTimeVariance,totalFieldVarianceInVoxelSpace,totalFieldTimeVariance] TimeVaryingBSplineVelocityField[gradientStep,velocityFieldMeshSize,<numberOfTimePointSamples=4>,<splineOrder=3>] SyN[gradientStep,<updateFieldVarianceInVoxelSpace=3>,<totalFieldVarianceInVoxelSpace=0>] BSplineSyN[gradientStep,updateFieldMeshSizeAtBaseLevel,<totalFieldMeshSizeAtBaseLevel=0>,<splineOrder=3>] Exponential[gradientStep,updateFieldVarianceInVoxelSpace,velocityFieldVarianceInVoxelSpace,<numberOfIntegrationSteps>] BSplineExponential[gradientStep,updateFieldMeshSizeAtBaseLevel,<velocityFieldMeshSizeAtBaseLevel=0>,<numberOfIntegrationSteps>,<splineOrder=3>] Several transform options are available. The gradientStep or learningRate characterizes the gradient descent optimization and is scaled appropriately for each transform using the shift scales estimator. Subsequent parameters are transform-specific and can be determined from the usage. For the B-spline transforms one can also specify the smoothing in terms of spline distance (i.e. knot spacing). -c, --convergence MxNxO [MxNxO,<convergenceThreshold=1e-6>,<convergenceWindowSize=10>] Convergence is determined from the number of iterations per level and is determined by fitting a line to the normalized energy profile of the last N iterations (where N is specified by the window size) and determining the slope which is then compared with the convergence threshold. -s, --smoothing-sigmas MxNxO... Specify the sigma of gaussian smoothing at each level. Units are given in terms of voxels ('vox') or physical spacing ('mm'). Example usage is '4x2x1mm' and '4x2x1vox' where no units implies voxel spacing. -f, --shrink-factors MxNxO... Specify the shrink factor for the virtual domain (typically the fixed image) at each level. -u, --use-histogram-matching Histogram match the images before registration. -l, --use-estimate-learning-rate-once turn on the option that lets you estimate the learning rate step size only at the beginning of each level. * useful as a second stage of fine-scale registration. -w, --winsorize-image-intensities [lowerQuantile,upperQuantile] Winsorize data based on specified quantiles. -x, --masks [fixedImageMask,movingImageMask] Image masks to limit voxels considered by the metric. Two options are allowed for mask specification: 1) Either the user specifies a single mask to be used for all stages or 2) the user specifies a mask for each stage. With the latter one can select to which stages masks are applied by supplying valid file names. If the file does not exist, a mask will not be used for that stage. Note that we handle the fixed and moving masks separately to enforce this constraint. --float Use 'float' instead of 'double' for computations. <VALUES>: 0 --minc Use MINC file formats for transformations. <VALUES>: 0 --random-seed seedValue Use a fixed seed for random number generation. By default, the system clock is used to initialize the seeding. The fixed seed can be any nonzero int value. -v, --verbose (0)/1 Verbose output. -h Print the help menu (short version). --help Print the help menu. Will also print values used on the current command line call. <VALUES>: 1antsJointLabelFusion.shrelies onsbatch,revandcutto limit memory:Other leads: maybe use
LegacyMultiProc,nipreps/fmriprep#836,
nipreps/fmriprep#839,
nipreps/fmriprep#854,
nipy/nipype#2284 (
maxtasksperchild 1? https://nipype.readthedocs.io/en/latest/api/generated/nipype.pipeline.plugins.legacymultiproc.html), nipy/nipype#2548, nipy/nipype#2773The text was updated successfully, but these errors were encountered: