Skip to content

Commit

Permalink
Fix permute of 4D data (mainly MMS FPI cdf files)
Browse files Browse the repository at this point in the history
After a alot of testing, I am convinced that the previous permute() introduced back in commit fec5c15 is incorrect for files created according to the most recent instructions from NASA SPDF (comments in spdfcdfwrite.m).
As such the MMS FPI files have previously been permuted incorrectly in irfu-matlab for 4D data variables. With this commit we will still permute() to ensure that the time dependency is the first column, files are written with time dependency as last column, but we will not permute() other columns unless they are created according to the old style (Cluster, etc). This will better ensure that the "Depend_i" variables correctly correspond to the data in dataobj.

A longer and more detailed overview of some of the tests performed for this change can be viewed here: https://docs.google.com/spreadsheets/d/1L58nLwy1Mwl9y4OATjkgjb58p6dCiNFerEx2cVbsB0M/edit#gid=0

Also see messages of the previous commits 3b09811 and a01031a
  • Loading branch information
thomas-nilsson-irfu committed Dec 7, 2017
1 parent 9e25aeb commit 9440568
Showing 1 changed file with 21 additions and 37 deletions.
58 changes: 21 additions & 37 deletions @dataobj/dataobj.m
Original file line number Diff line number Diff line change
Expand Up @@ -355,49 +355,33 @@
end

function fix_order_of_array_dimensions
% oldPermute=true;
% if oldPermute
for iDimension=3:4 %#ok<UNRCH>
indDatasets=find(cellfun(@(x) numel(size(x)),data(:))==iDimension); % find iDimension datasets
for iDimension=3:4
% Check if dimensions match expected (ignoring Depend_0) and that
% it is not a single vector (which could be "Nx1").
indDatasets = find(cellfun(@(x) all(x(:) > 1) && numel(x)==iDimension-1, info.Variables(:,2)));
for iDataset=1:numel(indDatasets)
if iDimension==3
data{indDatasets(iDataset)}=permute(data{indDatasets(iDataset)},[3 1 2]);
% permutate it to N x m x n, where N corrsponds to record (ie Depend_0).
data{indDatasets(iDataset)} = permute(data{indDatasets(iDataset)}, [3 1 2]);
elseif iDimension==4
data{indDatasets(iDataset)}=permute(data{indDatasets(iDataset)},[4 3 1 2]);
% Check if it should be simple cyclic permutation (MMS), data
% was read into "recSize x nRec". Also possibly a matrix of
% size "recSize" if it only contained one single record.
% Or if the data should be more complexly permuted (Cluster),
% ie where the recSize does not match the data.
recSize = info.Variables{indDatasets(iDataset), 2}; % Expected record size, (m x n x p)
nRec = info.Variables{indDatasets(iDataset), 3}; % Numer of records, (N)
dataSize = size(data{indDatasets(iDataset)}); % Size of data, as it was read
if isequal([recSize, nRec], dataSize) || (nRec==1 && isequal(recSize, dataSize) )
% Simple cyclic permutation to get N x m x n x p
data{indDatasets(iDataset)} = permute(data{indDatasets(iDataset)}, [4 1 2 3]);
else
% Re-order Cluster data, from (n x p x m x N) to (N x m x n x p).
data{indDatasets(iDataset)} = permute(data{indDatasets(iDataset)}, [4 3 1 2]);
end
end
end
end
% else
% % NEW METHOD; TO BE TESTED
% for iDimension=3:4 %#ok<UNRCH>
% % Check if dimensions match expected (ignoring Depend_0) and that
% % it is not a single vector (which could be "Nx1").
% indDatasets = find(cellfun(@(x) all(x(:) > 1) && numel(x)==iDimension-1, info.Variables(:,2)));
% for iDataset=1:numel(indDatasets)
% if iDimension==3
% % permutate it to N x m x n, where N corrsponds to record (ie Depend_0).
% data{indDatasets(iDataset)} = permute(data{indDatasets(iDataset)}, [3 1 2]);
% elseif iDimension==4
% % Check if it should be simple cyclic permutation (MMS), data
% % was read into "recSize x nRec". Also possibly a matrix of
% % size "recSize" if it only contained one single record.
% % Or if the data should be more complexly permuted (Cluster),
% % ie where the recSize does not match the data.
% recSize = info.Variables{indDatasets(iDataset), 2}; % Expected record size, (m x n x p)
% nRec = info.Variables{indDatasets(iDataset), 3}; % Numer of records, (N)
% dataSize = size(data{indDatasets(iDataset)}); % Size of data, as it was read
% if isequal([recSize, nRec], dataSize) || (nRec==1 && isequal(recSize, dataSize) )
% % Simple cyclic permutation to get N x m x n x p
% data{indDatasets(iDataset)} = permute(data{indDatasets(iDataset)}, [4 1 2 3]);
% else
% % Re-order Cluster data, from (n x p x m x N) to (N x m x n x p).
% data{indDatasets(iDataset)} = permute(data{indDatasets(iDataset)}, [4 3 1 2]);
% end
% end
% end
% end
% % END OF NEW METHOD
% end
end

function update_variable_attributes_time % nested function
Expand Down

0 comments on commit 9440568

Please sign in to comment.