Skip to content

Commit

Permalink
jsonlab 0.8.0, major speed up of loadjson for matlab
Browse files Browse the repository at this point in the history 
git-svn-id: http://svn.code.sf.net/p/iso2mesh/code/trunk/jsonlab@338 786e58fb-9377-0410-9ff7-e4ac0ac0635c
  • Loading branch information
fangq committed Jan 14, 2012
1 parent 49cd02a commit 5b82753
Show file tree
Hide file tree
Showing 4 changed files with 148 additions and 74 deletions.
11 changes: 9 additions & 2 deletions ChangeLog.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,19 @@

JSONlab ChangeLog (key features marked by *):

== JSONlab 0.5.1 (codename: Nexus Update 1), FangQ <fangq nmr mgh harvard edu> ==
== JSONlab 0.8.0 (codename: Sentiel), FangQ <fangq (at) nmr.mgh.harvard.edu> ==

2012/01/13 *speed up loadjson by 20 fold when parsing large data arrays in matlab
2012/01/11 remove row bracket if an array has 1 element, suggested by Mykel Kochenderfer
2011/12/22 *accept sequence of 'param',value input in savejson and loadjson
2011/11/18 fix struct array bug reported by Mykel Kochenderfer

== JSONlab 0.5.1 (codename: Nexus Update 1), FangQ <fangq (at) nmr.mgh.harvard.edu> ==

2011/10/21 fix a bug in loadjson, previous code does not use any of the acceleration
2011/10/20 loadjson supports JSON collections: appendable JSON objects

== JSONlab 0.5.0 (codename: Nexus), FangQ <fangq nmr mgh harvard edu> ==
== JSONlab 0.5.0 (codename: Nexus), FangQ <fangq (at) nmr.mgh.harvard.edu> ==

2011/10/16 package and release jsonlab 0.5.0
2011/10/15 *add json demo and regression test, support cpx numbers, fix double quote bug
Expand Down
137 changes: 75 additions & 62 deletions README.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
= An open-source MATLAB/Octave JSON encoder and decoder =
===============================================================================

*Copyright (c) 2011 Qianqian Fang <fangq at nmr.mgh.harvard.edu>
*Copyright (c) 2011,2012 Qianqian Fang <fangq at nmr.mgh.harvard.edu>
*License: BSD or GNU General Public License version 3 (GPL v3), see License*.txt
*Version: 0.5.1 (Nexus)
*Version: 0.8.0 (Sentinel)

-------------------------------------------------------------------------------

Expand Down Expand Up @@ -65,70 +65,82 @@ the two functions can be found below:

=== loadjson.m ===
<pre>
data=loadjson(fname)

parse a JSON (JavaScript Object Notation) file or string

authors:Qianqian Fang (fangq<at> nmr.mgh.harvard.edu)
date: 2011/09/09
Nedialko Krouchev: http://www.mathworks.com/matlabcentral/fileexchange/25713
date: 2009/11/02
François Glineur: http://www.mathworks.com/matlabcentral/fileexchange/23393
date: 2009/03/22
Joel Feenstra: http://www.mathworks.com/matlabcentral/fileexchange/20565
date: 2008/07/03

input:
fname: input file name, if fname contains "{}" or "[]", fname
will be interpreted as a JSON string

output:
dat: a cell array, where {...} blocks are converted into cell arrays,
and [...] are converted to arrays
data=loadjson(fname,opt)
or
data=loadjson(fname,'param1',value1,'param2',value2,...)

parse a JSON (JavaScript Object Notation) file or string

authors:Qianqian Fang (fangq<at> nmr.mgh.harvard.edu)
date: 2011/09/09
Nedialko Krouchev: http://www.mathworks.com/matlabcentral/fileexchange/25713
date: 2009/11/02
Fran�ois Glineur: http://www.mathworks.com/matlabcentral/fileexchange/23393
date: 2009/03/22
Joel Feenstra: http://www.mathworks.com/matlabcentral/fileexchange/20565
date: 2008/07/03

input:
fname: input file name, if fname contains "{}" or "[]", fname
will be interpreted as a JSON string
opt: a struct to store parsing options, opt can be replaced by
a list of ('param',value) pairs. The param string is equivallent
to a field in opt.

output:
dat: a cell array, where {...} blocks are converted into cell arrays,
and [...] are converted to arrays
</pre>

=== savejson.m ===

<pre>
json=savejson(rootname,obj,opt)

convert a MATLAB object (cell, struct or array) into a JSON (JavaScript
Object Notation) string

authors:Qianqian Fang (fangq<at> nmr.mgh.harvard.edu)
date: 2011/09/09

input:
rootname: name of the root-object, if set to '', will use variable name
obj: a MATLAB object (array, cell, cell array, struct, struct array)
opt: a struct for additional options, use [] if all use default
opt can have the following fields (first in [.|.] is the default)
opt.FloatFormat ['%.10g'|string]: format to show each numeric element
of a 1D/2D array;
opt.ArrayIndent [1|0]: if 1, output explicit data array with
precedent indentation; if 0, no indentation
opt.ArrayToStruct[0|1]: when set to 0, savejson outputs 1D/2D
array in JSON array format; if sets to 1, an
array will be shown as a struct with fields
"_ArrayType_", "_ArraySize_" and "_ArrayData_"; for
sparse arrays, the non-zero elements will be
saved to _ArrayData_ field in triplet-format i.e.
(ix,iy,val) and "_ArrayIsSparse_" will be added
with a value of 1; for a complex array, the
_ArrayData_ array will include two columns
(4 for sparse) to record the real and imaginary
parts, and also "_ArrayIsComplex_":1 is added.
opt.ParseLogical [0|1]: if this is set to 1, logical array elem
will use true/false rather than 1/0.

output:
json: a string in the JSON format (see http://json.org)

examples:
a=struct('node',[1 9 10; 2 1 1.2], 'elem',[9 1;1 2;2 3],...
'face',[9 01 2; 1 2 3; NaN,Inf,-Inf], 'author','FangQ');
savejson('mesh',a)
savejson('',a,struct('ArrayIndent',0,'FloatFormat','\t%.5g'))
json=savejson(rootname,obj,opt)
or
json=savejson(rootname,obj,'param1',value1,'param2',value2,...)

convert a MATLAB object (cell, struct or array) into a JSON (JavaScript
Object Notation) string

author: Qianqian Fang (fangq<at> nmr.mgh.harvard.edu)
created on 2011/09/09

input:
rootname: name of the root-object, if set to '', will use variable name
obj: a MATLAB object (array, cell, cell array, struct, struct array)
opt: a struct for additional options, use [] if all use default
opt can have the following fields (first in [.|.] is the default)
opt.FloatFormat ['%.10g'|string]: format to show each numeric element
of a 1D/2D array;
opt.ArrayIndent [1|0]: if 1, output explicit data array with
precedent indentation; if 0, no indentation
opt.ArrayToStruct[0|1]: when set to 0, savejson outputs 1D/2D
array in JSON array format; if sets to 1, an
array will be shown as a struct with fields
"_ArrayType_", "_ArraySize_" and "_ArrayData_"; for
sparse arrays, the non-zero elements will be
saved to _ArrayData_ field in triplet-format i.e.
(ix,iy,val) and "_ArrayIsSparse_" will be added
with a value of 1; for a complex array, the
_ArrayData_ array will include two columns
(4 for sparse) to record the real and imaginary
parts, and also "_ArrayIsComplex_":1 is added.
opt.ParseLogical [0|1]: if this is set to 1, logical array elem
will use true/false rather than 1/0.
opt.NoRowBracket [1|0]: if this is set to 1, arrays with a single
numerical element will be shown without a square
bracket, unless it is the root object; if 0, square
brackets are forced for any numerical arrays.
opt can be replaced by a list of ('param',value) pairs. The param
string is equivallent to a field in opt.
output:
json: a string in the JSON format (see http://json.org)

examples:
a=struct('node',[1 9 10; 2 1 1.2], 'elem',[9 1;1 2;2 3],...
'face',[9 01 2; 1 2 3; NaN,Inf,-Inf], 'author','FangQ');
savejson('mesh',a)
savejson('',a,'ArrayIndent',0,'FloatFormat','\t%.5g')
</pre>

=== examples ===
Expand All @@ -155,7 +167,8 @@ Here are the known issues:
# When processing names containing multi-byte characters, Octave and MATLAB \
can give different field-names;
# Can not handle classes;
# Although significantly accelerated, loadjson of large JSON file may still take some time.
# Although significantly accelerated, running loadjson for large JSON file may \
still take some time.

-------------------------------------------------------------------------------

Expand Down
55 changes: 50 additions & 5 deletions loadjson.m
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,12 @@
% date: 2009/11/02
% François Glineur: http://www.mathworks.com/matlabcentral/fileexchange/23393
% date: 2009/03/22
% Joel Feenstra: http://www.mathworks.com/matlabcentral/fileexchange/20565
% Joel Feenstra:
% http://www.mathworks.com/matlabcentral/fileexchange/20565
% date: 2008/07/03
%
% $Id$
%
% input:
% fname: input file name, if fname contains "{}" or "[]", fname
% will be interpreted as a JSON string
Expand Down Expand Up @@ -177,14 +180,50 @@
global pos inStr isoct
parse_char('[');
object = cell(0, 1);
dim2=[];
if next_char ~= ']'
endpos=matching_bracket(inStr,pos);
[endpos e1l e1r maxlevel]=matching_bracket(inStr,pos);
arraystr=['[' inStr(pos:endpos)];
arraystr=regexprep(arraystr,'"_NaN_"','NaN');
arraystr=regexprep(arraystr,'"([-+]*)_Inf_"','$1Inf');
arraystr(find(arraystr==sprintf('\n')))=[];
arraystr(find(arraystr==sprintf('\r')))=[];
%arraystr=regexprep(arraystr,'\s*,',','); % this is slow,sometimes needed
if(~isempty(e1l) && ~isempty(e1r)) % the array is in 2D or higher D
astr=inStr((e1l+1):(e1r-1));
astr=regexprep(astr,'"_NaN_"','NaN');
astr=regexprep(astr,'"([-+]*)_Inf_"','$1Inf');
astr(find(astr==sprintf('\n')))=[];
astr(find(astr==sprintf('\r')))=[];
astr(find(astr==' '))='';
if(isempty(find(astr=='[', 1))) % array is 2D
dim2=length(sscanf(astr,'%f,',[1 inf]));
end
else % array is 1D
astr=arraystr(2:end-1);
astr(find(astr==' '))='';
[obj count errmsg nextidx]=sscanf(astr,'%f,',[1,inf]);
if(nextidx>=length(astr)-1)
object=obj;
pos=endpos;
parse_char(']');
return;
end
end
if(~isempty(dim2))
astr=arraystr;
astr(find(astr=='['))='';
astr(find(astr==']'))='';
astr(find(astr==' '))='';
[obj count errmsg nextidx]=sscanf(astr,'%f,',inf);
if(nextidx>=length(astr)-1)
object=reshape(obj,dim2,numel(obj)/dim2)';
pos=endpos;
parse_char(']');
return;
end
end
arraystr=regexprep(arraystr,'\]\s*,','];');
arraystr=regexprep(arraystr,'"_NaN_"','NaN');
arraystr=regexprep(arraystr,'"([-+]*)_Inf_"','$1Inf');
try
if(isoct && regexp(arraystr,'"','once'))
error('Octave eval can produce empty cells for JSON-like input');
Expand Down Expand Up @@ -415,25 +454,31 @@ function error_pos(msg)
end

%%-------------------------------------------------------------------------
function endpos = matching_bracket(str,pos)
function [endpos e1l e1r maxlevel] = matching_bracket(str,pos)
global arraytoken
level=1;
maxlevel=level;
endpos=0;
bpos=arraytoken(arraytoken>=pos);
tokens=str(bpos);
len=length(tokens);
pos=1;
e1l=[];
e1r=[];
while(pos<=len)
c=tokens(pos);
if(c==']')
level=level-1;
if(isempty(e1r)) e1r=bpos(pos); end
if(level==0)
endpos=bpos(pos);
return
end
end
if(c=='[')
if(isempty(e1l)) e1l=bpos(pos); end
level=level+1;
maxlevel=max(maxlevel,level);
end
if(c=='"')
pos=matching_quote(tokens,pos+1);
Expand Down
19 changes: 14 additions & 5 deletions savejson.m
View file
Original file line number Diff line number Diff line change
@@ -1,12 +1,16 @@
function json=savejson(rootname,obj,varargin)
%
% json=savejson(rootname,obj,opt)
% or
% json=savejson(rootname,obj,'param1',value1,'param2',value2,...)
%
% convert a MATLAB object (cell, struct or array) into a JSON (JavaScript
% Object Notation) string
%
% authors:Qianqian Fang (fangq<at> nmr.mgh.harvard.edu)
% date: 2011/09/09
% author: Qianqian Fang (fangq<at> nmr.mgh.harvard.edu)
% created on 2011/09/09
%
% $Id$
%
% input:
% rootname: name of the root-object, if set to '', will use variable name
Expand All @@ -30,15 +34,20 @@
% parts, and also "_ArrayIsComplex_":1 is added.
% opt.ParseLogical [0|1]: if this is set to 1, logical array elem
% will use true/false rather than 1/0.
%
% opt.NoRowBracket [1|0]: if this is set to 1, arrays with a single
% numerical element will be shown without a square
% bracket, unless it is the root object; if 0, square
% brackets are forced for any numerical arrays.
% opt can be replaced by a list of ('param',value) pairs. The param
% string is equivallent to a field in opt.
% output:
% json: a string in the JSON format (see http://json.org)
%
% examples:
% a=struct('node',[1 9 10; 2 1 1.2], 'elem',[9 1;1 2;2 3],...
% 'face',[9 01 2; 1 2 3; NaN,Inf,-Inf], 'author','FangQ');
% savejson('mesh',a)
% savejson('',a,struct('ArrayIndent',0,'FloatFormat','\t%.5g'))
% savejson('',a,'ArrayIndent',0,'FloatFormat','\t%.5g')
%
% license:
% BSD or GPL version 3, see LICENSE_{BSD,GPLv3}.txt files for details
Expand Down Expand Up @@ -203,7 +212,7 @@
txt=sprintf('%s%s',padding1,matdata2json(item,level+1,varargin{:}));
else
if(numel(item)==1 && jsonopt('NoRowBracket',1,varargin{:})==1)
numtxt=regexprep(regexprep(matdata2json(item,level+1,varargin{:}),'^[',''),']','');
numtxt=regexprep(regexprep(matdata2json(item,level+1,varargin{:}),'^\[',''),']','');
txt=sprintf('%s"%s": %s',padding1,name,numtxt);
else
txt=sprintf('%s"%s": %s',padding1,name,matdata2json(item,level+1,varargin{:}));
Expand Down

0 comments on commit 5b82753

Please sign in to comment.