Add hash mechanism and speed up process large ASN.1 files

[brchiu]

Add hash mechanism and modify code to speed up processing of large ASN.1 file.

Before applying this PR, it takes over 75 minutes to process rrc-7.1.0.asn1

Makefile generation finished

real	75m51.124s
user	75m47.927s
sys	0m0.436s

and after applying this PR, it takes less than 2 minutes.

Makefile generation finished

real	1m23.242s
user	1m21.253s
sys	0m0.495s

The reason that it's so slow to processing large ASN.1 file is we have many duplicate check mechanism implemented in various development stages. And name lookup in imported modules need to be done.

We can continue this improvement of processing time by combining duplicate check in future.
And there might be other places can be speed up by introducing hash mechanism.

[coveralls]

Coverage increased (+0.1%) to 71.447% when pulling 531a154 on brchiu:add_hash_mechanism_and_speed_up_process into 3f52df0 on vlm:master.

[brchiu]

For people who is interesting in typical profiling result.
See the following two gists :

gprof result of asn1c (with pull request 237) processing rrc-7.1.0.asn1
gprof result of asn1c (with pull request 237) processing ranap-14.1.0.asn1

[vlm]

Implemented slightly differently, but thanks for prompting!

On rrc-7.1.0:

real	0m21.479s
user	0m21.240s
sys	0m0.160s

[velichkov]

Hi @vlm,

The asn1c crashes while compiling the S1AP with your latest master 08661a6 and it looks like the problem was introduced with in 1a49ced as with the previous commit it doesn't crash. Also it doesn't crash with the latest commit from this branch.

(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /home/vasko/sources/asn1c/asn1c/asn1c -S ../../skeletons -pdu=S1AP-PDU -fcompound-names -fno-include-deps -gen-PER -findirect-choice -flink-skeletons ../s1ap-14.4.0.asn1

Program received signal SIGSEGV, Segmentation fault.
_asn1f_assign_cell_value (counter=0, 
    bend=0x85e829 "SUCCESSFUL OUTCOME\t\tHandoverCommand\n\tUNSUCCESSFUL OUTCOME\tHandoverPreparationFailure\n\tPROCEDURE CODE\t\t\tid-HandoverPreparation\n\tCRITICALITY\t\t\t\treject\n}", buf=<optimized out>, cell=0x8677d0, arg=0x7fffffffd650) at asn1fix_cws.c:491
491			expr->reference = type_expr->reference;
(gdb) bt
#0  _asn1f_assign_cell_value (counter=0, 
    bend=0x85e829 "SUCCESSFUL OUTCOME\t\tHandoverCommand\n\tUNSUCCESSFUL OUTCOME\tHandoverPreparationFailure\n\tPROCEDURE CODE\t\t\tid-HandoverPreparation\n\tCRITICALITY\t\t\t\treject\n}", buf=<optimized out>, cell=0x8677d0, arg=0x7fffffffd650) at asn1fix_cws.c:491
#1  _asn1f_parse_class_object_data (arg=arg@entry=0x7fffffffd650, eclass=eclass@entry=0x690880, row=row@entry=0x695510, syntax=0x6912c0, 
    buf=<optimized out>, bend=0x85e8be "}", optional_mode=0, newpos=0x0, counter=0) at asn1fix_cws.c:361
#2  0x000000000041f040 in asn1f_check_class_object (arg=arg@entry=0x7fffffffd650) at asn1fix_cws.c:43
#3  0x0000000000418755 in constraint_object_resolve (value=<optimized out>, arg=0x7fffffffd650) at asn1fix_constraint.c:392
#4  asn1constraint_resolve (arg=arg@entry=0x7fffffffd650, ct=0x696340, etype=etype@entry=A1TC_CLASSDEF, 
    effective_type=effective_type@entry=ACT_INVALID) at asn1fix_constraint.c:219
#5  0x00000000004181a9 in asn1constraint_resolve (arg=arg@entry=0x7fffffffd650, ct=0x6964e0, etype=etype@entry=A1TC_CLASSDEF, 
    effective_type=effective_type@entry=ACT_INVALID) at asn1fix_constraint.c:227
#6  0x00000000004181a9 in asn1constraint_resolve (arg=arg@entry=0x7fffffffd650, ct=ct@entry=0x697e00, etype=A1TC_CLASSDEF, 
    effective_type=effective_type@entry=ACT_INVALID) at asn1fix_constraint.c:227
#7  0x0000000000418b32 in asn1constraint_pullup (arg=arg@entry=0x7fffffffd650) at asn1fix_constraint.c:74
#8  0x0000000000418518 in constraint_type_resolve (ct=0x695c30, arg=0x7fffffffdb50) at asn1fix_constraint.c:300
#9  asn1constraint_resolve (arg=arg@entry=0x7fffffffdb50, ct=0x695c30, etype=etype@entry=A1TC_CLASSDEF, 
    effective_type=effective_type@entry=ACT_INVALID) at asn1fix_constraint.c:201
#10 0x00000000004181a9 in asn1constraint_resolve (arg=arg@entry=0x7fffffffdb50, ct=0x695ee0, etype=etype@entry=A1TC_CLASSDEF, 
    effective_type=effective_type@entry=ACT_INVALID) at asn1fix_constraint.c:227
#11 0x00000000004181a9 in asn1constraint_resolve (arg=arg@entry=0x7fffffffdb50, ct=0x695fd0, etype=etype@entry=A1TC_CLASSDEF, 
    effective_type=effective_type@entry=ACT_INVALID) at asn1fix_constraint.c:227
#12 0x0000000000412b92 in asn1f_resolve_constraints (arg=0x7fffffffdb50) at asn1fix.c:433
#13 0x0000000000414342 in asn1f_recurse_expr (arg=arg@entry=0x7fffffffdb50, callback=callback@entry=0x412b20 <asn1f_resolve_constraints>)
    at asn1fix_misc.c:209
#14 0x0000000000412a72 in phase_1_1 (arg=arg@entry=0x7fffffffdb50, prm2=prm2@entry=0) at asn1fix.c:356
#15 0x000000000041334f in asn1f_fix_module__phase_1 (arg=0x7fffffffdb50) at asn1fix.c:189
#16 asn1f_process (asn=asn@entry=0x6a6700, flags=<optimized out>, flags@entry=A1F_NOFLAGS, error_logger=0x412230 <_default_error_logger>, 
    error_logger@entry=0x0) at asn1fix.c:84
#17 0x0000000000402164 in main (ac=1, av=0x7fffffffde10) at asn1c.c:328
(gdb) list
486		if(expr->reference &&
487			!asn1f_lookup_symbol(arg, expr->rhs_pspecs, expr->reference)) {
488	
489			asn1p_ref_free(expr->reference);
490			new_ref = 0;
491			expr->reference = type_expr->reference;
492			if (asn1f_value_resolve(arg, expr, 0)) {
493				expr->reference = 0;
494				asn1p_expr_free(expr);
495				FATAL("Cannot find %s referenced by %s at line %d",
(gdb) p type_expr
$1 = (asn1p_expr_t *) 0x0

[brchiu]

Sorry for my previous wrong finding. Please ignore that message.

Below is the correct reason and solution of crash :

Line 66 and 67 of asn1f_lookup_in_imports() in asn1fix_retrieve.c : 

            asn1p_expr_t *v = v =
                asn1f_lookup_in_module(fromModule, tc->Identifier);

Should be :

            asn1p_expr_t *v =
                asn1f_lookup_in_module(fromModule, name);

[velichkov]

            asn1f_lookup_in_module(fromModule, name);

Yeah, this fixes the crash!

[brchiu]

For larger ASN.1 file, e.g. nbap-14.1.0 of 1167826 bytes compared with rrc-7.1.0.asn1 of 723689 bytes, though it can not be successfully processed because of issue,

@vlm's hash implementation takes :

FATAL: Unsupported type ProcedureID for value _procedureCode_id_cellSetup_ddMode_fdd_
FATAL: Cannot compile "InitiatingMessage" (20:1) at line 242
FATAL: Cannot compile "InitiatingMessage" (20:1) at line 242
Makefile:13: recipe for target 'regen-makefile' failed
make: *** [regen-makefile] Error 70

real	26m22.332s
user	26m21.647s
sys	0m0.162s

and this pull request implementation takes :

FATAL: Unsupported type ProcedureID for value _procedureCode_id_cellSetup_ddMode_fdd_
FATAL: Cannot compile "InitiatingMessage" (20:1) at line 242
FATAL: Cannot compile "InitiatingMessage" (20:1) at line 242
Makefile:13: recipe for target 'regen-makefile' failed
make: *** [regen-makefile] Error 70

real	0m12.930s
user	0m12.619s
sys	0m0.121s

Perhaps there still be something worthy improved in @vlm's hash implementation.
See gprof result without stand library info, this profiling result does not show time spending in function calls of standard library, e.g. strcmp(), which might be relevant in this case.