Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade Nmap's PCRE library (libpcre) to pcre2 #1335

Open
cpaelzer opened this issue Sep 26, 2018 · 12 comments
Open

Upgrade Nmap's PCRE library (libpcre) to pcre2 #1335

cpaelzer opened this issue Sep 26, 2018 · 12 comments

Comments

@cpaelzer
Copy link

Hi,
pcre2 was not mentioned commits nor on the mailing lists nor the issue tracker that now is mirrored to GH, unless I missed something.

pcre2 is available since early 2015 and projects could start considering it stable, but due to a change in the API it usually is more than just changing the include being used.
pcre (8.x) continues to exists, but as bugifx only stream and any new features/development are going into pcre2 (10.x).

Therefore Ubuntu would like to demote pcre in favor of pcre2 at some point, but Nmap (among other pkgs) would need to be ported to use pcre2.

Therefore I wanted to ask if there are any plans to do so (or not) in your project?
If there is none yet, please feel free to use this issue to track it.

@dmiller-nmap
Copy link

Thanks for suggesting this. The Nmap Project currently has no plans to update to PCRE2, but we have no immediate objections. However, these are our questions that would have to be answered before we commit to an upgrade:

  1. Will all of our tens of thousands of existing regular expressions match the same strings that they did under original PCRE?
  2. Does PCRE2 build and run on the platforms we need it to? Platforms officially supported by Nmap include GNU/Linux, Windows, Solaris, AIX, and FreeBSD, among others.
  3. Has performance been negatively impacted for our specific use case? We compile tens of thousands of regular expressions and then attempt to find the one that matches some untrusted input, for up to several thousand inputs in succession.

There may be other concerns, but these are the primary ones I can think of.

@cpaelzer
Copy link
Author

Hi,
thanks for your answer @dmiller-nmap all concerns sound very reasonable.
First of all I'm not an pcre expert at all, just trying to understand/coordinate various projects position to prce2 from a Distributions perspective.
Therefore I'm trying to answer your concerns to my best knowledge, but do not really feel authoritative to do so- upstream pcre might be the right place to check these I'd think.

  1. AFAICT the API changed but not the Rules. There are new constants that can be added to modify the behavior of rules but I'd expect the rules to match the same. News and Changelog are meant to give a high level overview and I found nothing suggesting that rules would behave differently.
  2. Sorry I don't know, but again I have not found anything in upstreams statements that ould sugegst that pcre2 would be inferior in that regard. Via search engines I found pcre2 builds for all platforms you named, so I'd think yes - the same platforms are supported.
  3. Well given that any new development since 2015 is in pcre2 I'd hope that if anything it got better there.

As I said, I'm the middle man in this case.
I'm moore "trying to track your intention and status" than "trying to convince" you on this.

@fyodor fyodor changed the title plans for pcre2 support? Upgrade Nmap's PCRE library (libpcre) to pcre2 Jun 20, 2022
@fyodor
Copy link
Member

fyodor commented Jun 20, 2022

Thanks for opening this issue (back in 2018), @cpaelzer. I agree that we should probably upgrade to pcre2. Especially now that pcre1 is end-of-life. The current version 8.45 is expected to be the final release. We were thinking of upgrading Nmap to pcre1 version 8.45 first (see Issue #2250), but I'm not sure there is much value in doing so. It may be better and easier to just jump directly to pcre2. I'm adjusting our roadmap accordingly.

@XWwalker
Copy link

XWwalker commented Jan 4, 2023

@bonsaiviking @fyodor Hi. Does nmap have any progress on upgrading PCRE2?

@david-geiger
Copy link

Hi,

Some distributions, like us Mageia Linux, now started to drop old pcre library from their mirrors.

Regards,
David

@XWwalker
Copy link

Hi, I have added a new PR that supports pcre2 compilation, please check and merge.

@hillu
Copy link

hillu commented Jul 2, 2023

Debian has marked all reverse dependencies on pcre3 as release-critical which means nmap will soon be auto-removed from the testing distribution. See Debian bug#1000012.

@fyodor
Copy link
Member

fyodor commented Jul 3, 2023

Thanks for the note, @hillu. We finished and checked in our PCRE2 upgrade last week, including testing on Linux, Mac, and Windows. It will be in the next Nmap release. We don't have an exact date scheduled for that release, but it will probably be in August or September. If Debian needs it before then, perhaps you could cherry-pick the changes from GitHub?

@hillu
Copy link

hillu commented Jul 7, 2023

If Debian needs it before then, perhaps you could cherry-pick the changes from GitHub?

I've done just that, thanks.

@yselkowitz
Copy link

RHEL 10 also plans on dropping support for classic pcre, and Fedora will certainly drop it in due course.

@iasdeoupxe
Copy link

  1. Will all of our tens of thousands of existing regular expressions match the same strings that they did under original PCRE?

There could be some compatibility problems depending on the used regex. For example this was valid PCRE:

^[[:alnum:]-_.]+$

which would need to be one of these in PCRE2 (hypen escaped or at the very first or last position):

^[[:alnum:]_.-]+$
^[-[:alnum:]_.]+$
^[[:alnum:]\-_.]+$

References:

@dmiller-nmap
Copy link

We updated the code back in 828ab48. We are keeping this issue open until we have the Windows and macOS builds confirmed working.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants