Optimize fields validation

[jsoriano]

While trying some changes in pipeline tests I found that testing some packages was taking much more time than I would have expected.

I have enabled profiling and I have found that most of the time, elastic-package is compiling regular expressions.

Refactoring the code to don't use regular expressions drastically improves the execution times. For example in the case of the panw module it goes in my laptop from more than 3 minutes to less than 10 seconds.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-03-22T18:32:13.689+0000

• Duration: 20 min 14 sec

Test stats 🧪

Test	Results
Failed	0
Passed	602
Skipped	1
Total	603

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[mtojek]

I admit that I didn't expect it to happen. Good catch!

[jsoriano]

I have refactored the code to don't need regular expressions, I see similar improvements in time than using a regular expressions cache.

[jsoriano]

I found profiling that strings.Split was also expensive (though nothing close to regexp compilation), I have further refactored to do char by char comparison, this reduced the testing time of the panw module to less than 6 seconds (~3 mins original code, ~10 seconds with regexp cache or with first iteration without regexps, ~6 seconds with second iteration).

Shared profile with current code:
cpu.profile.gz

[mtojek]

Nice optimization, although the code readability might be impaired a bit. Do you think that you can get rid of "regexp" dependency here at all?

Before merging this PR, I suggest releasing a new minor version, so we can easily roll back if this PR becomes problematic.

[mtojek]

Do you think that we need some unit tests as it looks like a custom logic?

[jsoriano]

I thought that we already had good coverage for this, but adding a unit test has helped me found some wrong cases with wildcards. Good idea, thanks 👍

[jsoriano]

Nice optimization, although the code readability might be impaired a bit.

Well, this is relative, others may also find counterintuitive to build regular expressions on the fly 🙂 In any case the code is quite localized inside a single function with well-defined expectations. I have added comments in case it helps.

Do you think that you can get rid of "regexp" dependency here at all?

The other case where we use regexp is to validate patterns defined in the field definition, we need regexp compilation there. We could consider caching these regular expressions if they are a problem, but I think they are used only once most of the times, I haven't found these compilations while profiling.
We may consider optimizing this in a future PR if we find this is problematic.

Before merging this PR, I suggest releasing a new minor version, so we can easily roll back if this PR becomes problematic.

👍

[jsoriano]

/test

[jsoriano]

/test

[jsoriano]

/test