Migrating from zlib to zlib-ng and concerns about compat build.

[netcorefan1]

This discussion helped me a lot and I would appreciate if you guys can help me to avoid making very bad things.

I was going to automatically override all vcpkg packages installation behavior by changing zlib dependency with zlib-ng COMPAT mode on the false belief that in this way they would use automatically next gen instructions when available and old zlib when not. From what I have understood, this would only happen if the source code of each package is modified to call functions from zlib-ng.h. Without this, it will just be like using the old zlib with high risks of crashes here and there. Is that correct?

Thank you both. So without re-compiling an application it is not recommended to use zlib-ng at all? I replaced the zlib1.dll from the game by the zlib-ng and it works. Have the feeling the loading times are a bit faster...

So, I'm wondering why I see here and elsewhere people trying to do a direct replacement when without recompiling and changing the source code it will just have the speed of the old zlib or a bit faster at best.
Or I am wrong and even compat mode (zlib.h) uses all (or some) available next gen instructions?
How to know it is effectively running all, some of none ngi? Any verbose log that can be activated?

The compatibility version is mostly useful when you can't rebuild the binary using zlib. For anything that has source code available, one should use non-compatibility version and adapt the source codes instead.

In the case of vcpkg packages, they're always compiled locally prior to installation. I have yet to understand how to modify the source code during the process, but by giving a quick look, patches can be applied by installing in editable mode. By reading the PORTING help, seems to be that all is needed is just a zng_ prefix which make me think it would be easy like doing a simple "search and replace". How can this be automated? Is there some universal patcher that one can use?

I've also like to know if support for legacy zlib compatibility is going to be dropped in the future in order to save myself from wasting time and resources for nothing.

[Dead2]

You seem to be confused about what is compat api vs native api, or perhaps what an API is to begin with.

zlib-ng compiled in zlib compat mode results in a dll/so file that acts like it is a stock zlib, but it is the full zlib-ng under the hood, all speedups included.
zlib-ng compiled with its own api is very similar but it uses different function names to avoid linker conflicts, and we have cleaned away several old functions, macros, and are using updated datatypes, etc.

Those two are not interchangeable. An application uses the dll/so file from one or the other API.

• If you have a bunch of old applications that don't support the zlib-ng API (If you make a linux distro for example), they can all be compiled with zlib-ng in zlib compat mode and word without changes to the applications themselves.

• If you develop your own application, you can choose; stick with the legacy zlib API, change to zlib-ng API with less cruft, or detect both and choose either during compile.

• If you have an application that you do not have the source code for, you can attempt to replace the zlib1.dll (or similarly named) that comes with that application with the zlib-ng compat dll (See our Releases page for download links to pre-made dlls). This should work, but we cannot guarantee that it will, because we can not know whether the developers of that application have made any changes to their zlib1.dll, whether their version is really old, whether it uses special calling conventions, etc.

If you are looking for information about how to change your application to from using zlib API to zlib-ng API, you can have a look at the patch I have made for Nginx, it switches Nginx from using zlib API to zlib-ng API, but it is a bit crude since it does not do compile-time detection and choice. https://github.com/zlib-ng/patches/blob/master/nginx/1.18.0-zlib-ng.patch
Note that Nginx already supports using zlib-ng via the compat API, but if a linux distro wants to include stock zlib and zlib-ng with its native api, that would mean it would still only use the stock zlib. This patch makes it choose to use the zlib-ng api instead.

I hope that clears things up a bit.

[netcorefan1]

@Dead2 Many thanks for your support.
You're right. I come from the managed world where there aren't such concepts when linking libraries. Static, shared, dynamic, crt etc are giving to me very hard times!

zlib-ng compiled in zlib compat mode results in a dll/so file that acts like it is a stock zlib, but it is the full zlib-ng under the hood, all speedups included.

• If you have a bunch of old applications that don't support the zlib-ng API (If you make a linux distro for example), they can all be compiled with zlib-ng in zlib compat mode and word without changes to the applications themselves.

So, seems to be that I was on the right track. I had success with some libraries, but compilation failed with others. Could this be due to different configure options? Or it is expected to work in all cases?
EDIT: it was my fault (may be some cached stuff not cleaned) and it worked for all the libraries I had chance to try!

• If you develop your own application, you can choose; stick with the legacy zlib API, change to zlib-ng API with less cruft, or detect both and choose either during compile.

Yes, for my own application zlib-ng native is probably the best and easy choice. For other situation (ex: cloned repos) is probably better to recompile with a zlib-ng package compiled in compat mode.

If you have an application that you do not have the source code for, you can attempt to replace the zlib1.dll (or similarly named) that comes with that application with the zlib-ng compat dll (See our Releases page for download links to pre-made dlls). This should work, but we cannot guarantee that it will, because we can not know whether the developers of that application have made any changes to their zlib1.dll, whether their version is really old, whether it uses special calling conventions, etc.

I would say rarely, but not so rarely as I think. For example, I compile OpenCv with CUDA support and they have their own zlib (misteriously renamed to zlibwapi.dll). I would avoid any attempt to replace this, but now, in the same process, I have CUDA libs that loads zlibwapi.dll and OpenCv that loads zlib-ng compat. One day, another package that depends on zlib-ng native could be added. I hope that this will not become a letal mix!

zlib-ng compiled with its own api is very similar but it uses different function names to avoid linker conflicts, and we have cleaned away several old functions, macros, and are using updated datatypes, etc. If you are looking for information about how to change your application to from using zlib API to zlib-ng API, you can have a look at the patch I have made for Nginx, it switches Nginx from using zlib API to zlib-ng API, but it is a bit crude since it does not do compile-time detection and choice.

Thanks for the link. Mhhh... seems to be that there is much more behind the scenes than just adding a simple prefix. This brought me to the conclusion that if I expect to get all vcpkg packages I need automatically patched to use zlib-ng natively before the build process, than I'm afraid this will never work. A simple search & replace solution will probably not work and I would basically have to go into the same process and when I come up with a patch, that patch will only work for that package. Unless someone with the right knowledge developed an universal patcher, I'm afraid it would be better to stick with native for my own application and compat for third parties libraries.

Thanks again. Your response cleared up several things and, most of all, saved me from taking dead-end roads. I have now a plan in mind. I'm not sure it can work since there are still a couple of concerns, but I don't want to go out argument and I will open a new discussion.