MD event workspace can not handle more then 0.3G events

[abuts]

I am attempting to create large MD file in memory on the machine with 0.5Tb of RAM.

The script to build such file is presented below, but similar Horace script (also supplied for comparison) generates 4Gb events, writes 151Gb file and takes (70min+3min+160min) to complete (3 stages of Horace processing -- converting single workspace, calculating indexes, writing combined file)

Horace and Mantid script expected to generate the same results (if they not, it is a bug of its own which I am investigating independently) but in fact the results are different.

The Mantid script (single thread version has been used) takes ~(30min+85min) to complete (2 stages -- load nxspe and convertToMD) (very approximate, as single threaded, takes substantial memory and strongly depends on the load of the machine (changes with load and during conversion time)) but it generates ~0.3Gb events in workspace and writes 17Gb file.

If may be that the Matnid script sets different conversion from Horace (different bug) but there is an malloc messages generated during conversion:
tcmalloc: large alloc 3451092992 bytes == 0x7fd2cecd8000 @ 0x7fdb90c1ef66 0x7fdb829025c8 0x7fda8127a800

160Gb is also the approximate size of all contributing nxspe files.

Which make me think that it is rather the workspace bug.

Mantid congersion script:

rint 'start converting to SQW\n'
cur_ws='wsn';
md_ws='mdws4D';
saveTmpTo='/home/wkc26243/ProcessedData/'
MDWSF='';
TargetWSName='WS4DMerged';

ic=0;
file_nums1 = range(15654,15723);
file_nums2 = range(15733,15834);
file_nums  = file_nums1+file_nums2
for n in file_nums:
    ic = ic+1;

    inFile = 'MAP'+str(n)+'.nxspe';
    print 'processing file name: '+inFile+' step: #'+str(ic)+'/'+str(len(file_nums))
    LoadNXSPE(Filename=inFile,OutputWorkspace=cur_ws)       
    SetUB(Workspace=cur_ws,a='2.87',b='2.87',c='2.87',u='1,0,0',v='0,1,0')
    #SetGoniometer(cur_ws,Axis0='Psi,0,1,0,1');
    print 'Adding to md workspace '
    ConvertToMDEvents(InputWorkspace=cur_ws,OutputWorkspace=TargetWSName,QDimensions='Q3D',OverwriteExisting='0',QConversionScales='HKL',UsePreprocessedDetectors='1',MinValues='-50,-50,-50,-150',MaxValues='50,50,50,1300')


print 'Saving Final workspace'
SaveMD(TargetWSName,Filename=saveTmpTo+TargetWSName+'.nxs');

Horace Conversion script:

function hor1400K_script
% where the script should be run
demo_dir=pwd;


% Give locations of input files
indir=pwd;     % source directory of spe files
par_file='';
par_file=parse_path([indir '/Instrument/map_4to1_jul09.par']);     % detector parameter file
sqw_file=fullfile(indir,'fe_E1400_8K.sqw');        % output sqw file
data_source =sqw_file;

% Set incident energy, lattice parameters etc.
efix=1371.5;
emode=1;
alatt=[2.87,2.87,2.87];
angdeg=[90,90,90];
u=[1,0,0];
v=[0,1,0];
omega=0;dpsi=0;gl=0;gs=0;

% Create the list of file names and orientation angles
% G1
[spe_file,psi]=build_fnames(indir,15654:15664,35,1,45);
% G2
[spe_file,psi]=build_fnames(indir,15665:15710,0,-1,-45,spe_file,psi);
% G3
[spe_file,psi]=build_fnames(indir,15711:15723,-44.5,1,-32.5,spe_file,psi);
% G4
[spe_file,psi]=build_fnames(indir,15733:15739,-22.5,1,-16.5,spe_file,psi);
% G5
[spe_file,psi]=build_fnames(indir,15740:15800,-15.5,1,44.5,spe_file,psi);
% G6
[spe_file,psi]=build_fnames(indir,15801:15815,-31.5,1,-17.5,spe_file,psi);
% G7
[spe_file,psi]=build_fnames(indir,15816:15834,0,1,18,spe_file,psi);


% Create sqw file
gen_sqw (spe_file, par_file, sqw_file, efix, emode, alatt, angdeg,...
         u, v, psi, omega, dpsi, gl, gs);

[abuts]

This issue was originally trac ticket 5853