Handling of zlib fwrite dependency #3872

MichaelChirico · 2019-09-14T04:32:44Z

#include <zlib.h> might fail on some machines. In this case it would be helpful to fail more helpfully.

Currently on rocker/r-ver:3.5.3 (e.g.), we get the following installation failure for dev data.table:

install.packages('data.table', type = 'source', repos='http://Rdatatable.github.io/data.table')
Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
trying URL 'http://Rdatatable.github.io/data.table/src/contrib/data.table_1.12.3.tar.gz'
Content type 'application/gzip' length 4960958 bytes (4.7 MB)
==================================================
downloaded 4.7 MB

* installing *source* package ‘data.table’ ...
** libs
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c assign.c -o assign.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c between.c -o between.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c bmerge.c -o bmerge.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c chmatch.c -o chmatch.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c cj.c -o cj.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c coalesce.c -o coalesce.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c dogroups.c -o dogroups.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c fastmean.c -o fastmean.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c fcast.c -o fcast.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c fifelse.c -o fifelse.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c fmelt.c -o fmelt.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c forder.c -o forder.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c frank.c -o frank.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c fread.c -o fread.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c freadR.c -o freadR.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c froll.c -o froll.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c frollR.c -o frollR.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c frolladaptive.c -o frolladaptive.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c fsort.c -o fsort.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c fwrite.c -o fwrite.o
fwrite.c:22:48: fatal error: zlib.h: No such file or directory
 #include "zlib.h"      // for writing gzip file
                                                ^
compilation terminated.
/usr/local/lib/R/etc/Makeconf:163: recipe for target 'fwrite.o' failed
make: *** [fwrite.o] Error 1
ERROR: compilation failed for package ‘data.table’
* removing ‘/usr/local/lib/R/site-library/data.table’
* restoring previous ‘/usr/local/lib/R/site-library/data.table’

The downloaded source packages are in
	‘/tmp/Rtmp7Po4yU/downloaded_packages’
Warning message:
In install.packages("data.table", type = "source", repos = "http://Rdatatable.github.io/data.table") :
  installation of package ‘data.table’ had non-zero exit status

(this doesn't affect the CRAN version since there's no fwrite+gz support there yet)

By contrast, here's how units package fails:

# install.packages('Rcpp') first on clean image
install.packages('units')
Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
trying URL 'https://mran.microsoft.com/snapshot/2019-04-26/src/contrib/units_0.6-2.tar.gz'
Content type 'application/octet-stream' length 917316 bytes (895 KB)
==================================================
downloaded 895 KB

* installing *source* package ‘units’ ...
** package ‘units’ successfully unpacked and MD5 sums checked
configure: units: 0.6-2
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking how to run the C++ preprocessor... g++ -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for stdbool.h that conforms to C99... no
checking for _Bool... no
checking for error_at_line... yes
checking for gcc... gcc
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for XML_ParserCreate in -lexpat... no
checking udunits2.h usability... no
checking udunits2.h presence... no
checking for udunits2.h... no
checking udunits2/udunits2.h usability... no
checking udunits2/udunits2.h presence... no
checking for udunits2/udunits2.h... no
checking for ut_read_xml in -ludunits2... no
configure: error: in `/tmp/Rtmp78wlOS/R.INSTALL3f75541cfe/units':
configure: error: 
--------------------------------------------------------------------------------
  Configuration failed because libudunits2.so was not found. Try installing:
    * deb: libudunits2-dev (Debian, Ubuntu, ...)
    * rpm: udunits2-devel (Fedora, EPEL, ...)
    * brew: udunits (OSX)
  If udunits2 is already installed in a non-standard location, use:
    --configure-args='--with-udunits2-lib=/usr/local/lib'
  if the library was not found, and/or:
    --configure-args='--with-udunits2-include=/usr/include/udunits2'
  if the header was not found, replacing paths with appropriate values.
  You can alternatively set UDUNITS2_INCLUDE and UDUNITS2_LIBS manually.
--------------------------------------------------------------------------------

See `config.log' for more details
ERROR: configuration failed for package ‘units’
* removing ‘/usr/local/lib/R/site-library/units’

The downloaded source packages are in
	‘/tmp/RtmpJB7VSi/downloaded_packages’
Warning message:
In install.packages("units") :
  installation of package ‘units’ had non-zero exit status

Quite useful & actionable!

I'm not sure how ubiquitous zlib is in the wild...

The text was updated successfully, but these errors were encountered:

MichaelChirico · 2019-09-14T04:36:10Z

It seems to do so we should add a configure script which can detect the missing zlib header:

configure in R-exts: https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Configure-and-cleanup
configure script used by units: https://github.com/r-quantities/units/blob/7da0e9fcf9d8e732a0ac372f69e6a1fb3357f7e2/configure

jangorecki · 2019-09-14T08:07:46Z

Could we compile without zlib support if it is not present? Then fwrite would error when attempting to use gz. This reduce OS software dependency, this is very important to stay lightweight.

st-pasha · 2019-09-14T19:22:46Z

We had a similar problem in (py) datatable, and ended up just shipping the zlib.h and zconf.h with the source distribution (h2oai/datatable#1957).

jangorecki · 2019-09-15T11:50:03Z

@st-pasha way of shipping zlib together with data.table is another option. Definitely better than current error. Yet IMO it is better to not ship it together, but escape support of gz fwrite when zlib was not present during compilation. I can imagine that not every user uses fwrite, and among them not many uses gz, ultimately zlib might be needed by a tiny fraction of data.table users.

MichaelChirico · 2019-09-16T10:23:19Z

Agree w Jan it's best if we can essentially make zlib a "Suggests" (optional), I just don't know how to do that at the C level. Seems the way we handle openmp could be a template but I couldn't quite make it work. Exploratory stuff here in case anyone sees how to take it over the line:

https://github.com/MichaelChirico/data.table/tree/zlib_suggests

philippechataignon · 2019-09-16T12:53:16Z

zlib is required for R : that's why I think that there wasn't a problem for gzip support. But you're right: the headers (zlib1g-dev on Debian/Ubuntu) are not always present and are needed for compiling. On Debian zlib1g-dev is a r-base-dev dependency: that's why it's often present. In the initial example, I had a look at the dockerfile and zlib1g-dev is in BUILDDEPS : why zlib header is not found a compile time ?

I don't think that including zlib headers is a good idea because distribution binaries from zlib and included headers in data.table may be incompatibles. In this case, data.table must include headers and source of the zlib library.

I think configure is the best idea to detect if zlib is present and compile without zlib if not. OK for suggest. A warning can be displayed at compile time like package units do (I think curl, git2r and xml2 act in similar way).

Another advantage of configure is that it could be possible to add another compression method like lz4 and xz (which need external libs too). I've tested lz4 and on my machine : lz4 fwrite is fastest than csv fwrite. xz is interesting to have better ratio compressed (never implemented).

I don't know how to write a configure script. If this solution is retained, I can try to see how to modify fwrite.c to use a ZLIB_PRESENT macro to have a fwrite version with or without gzip support.

ZLIB_PRESENT must be defined if zlib (lib and headers) is present on the system. If not, ZLIB_PRESENT is undefined and data.table is compiled without zlib support. Normally ZLIB_PRESENT (or an equivalent) will be defined (or not) by a configure script. See issue #3872.

HAVE_LIBZ must be defined if zlib (lib and headers) is present on the system. If not, HAVE_LIBZ is undefined and data.table is compiled without zlib support. Normally HAVE_LIBZ (or an equivalent) will be defined (or not) by a configure script (see ax_check_zlib). See issue Rdatatable#3872.

philippechataignon · 2019-09-16T21:41:31Z

This branch is an attempt to write a configure script to detect zlib. Feel free to test. Not a PR because it introduces changes in current data.table build process.

jangorecki · 2019-09-16T21:45:38Z

@philippechataignon any chance for a unit test? It has to be conditionally escaped of course.
Also it seems more appropriate to error than attempt to write non-gzip file.

philippechataignon · 2019-09-19T15:24:54Z

I can't go further on the branch fwrite_zlib_present et for me it's OK but perhaps too late/too dangerous for 1.12.4. I've replaced warning by stop as suggests. I've manually tested fwrite on data.table compiled with/without HAVE_ZLIB flag forced in fwrite.c and it's OK but I don't know how to write a unit test to verify the message error when there's no zlib headers on the machine.

MichaelChirico · 2019-09-19T16:32:50Z

@jangorecki I think this is something Gitlab builds would have to do... as I see it zlib is basically a suggested dependency. Already we nocov suggested R package dependencies... must be all the harder to do coverage tests for Suggests on C lib

@philippechataignon currently on rocker/r-ver:3.5.3 I see:

------------------------- ANTICONF ERROR ---------------------------
Configuration failed because zlib was not found. Try installing:
 * deb: zlib1g-dev (Debian, Ubuntu, etc)
 * rpm: zlib-devel (Fedora, CentOS, RHEL)
If zlib is already installed, check that 'pkg-config' is in your
PATH and PKG_CONFIG_PATH contains a zlib.pc file. If pkg-config
is unavailable you can set INCLUDE_DIR and LIB_DIR manually via:
R CMD INSTALL --configure-vars='INCLUDE_DIR=... LIB_DIR=...'
--------------------------------------------------------------------
ERROR: configuration failed for package ‘data.table’
* removing ‘/usr/local/lib/R/site-library/data.table’
Error in i.p(...) : 
  (converted from warning) installation of package ‘/tmp/RtmpZjnvPS/file124bbd766/data.table_1.12.3.tar.gz’ had non-zero exit status

Is it possible for this to be a warning & allow installation to continue?

I see (converted from warning) there but getOption('warn') is 0 on the session I'm installing from so that must be set from the installation script?

philippechataignon · 2019-09-19T19:40:16Z

There was a exit 1 in configure script when R_CONFIG_ERROR is set. I removed it. Please can you test docker build again ? Thanks.

mattdowle · 2019-09-19T22:11:06Z

Looks great. Agree it's a bit risky/late for 1.12.4 so postponing to next release as suggested.

jangorecki · 2019-09-20T11:26:57Z

By postponding to 1.13.0 we can also get insight if there are many affected users. Assuming that zlib should be installed as r-base-dev dependency then we should consider this situation as abnormal. IMO it might not be worth to introduce configure script just to cover that abnormal situation, unless of course it affects many users.
@MichaelChirico maybe worth to report this issue in rocker/r-ver:3.5.3?

MichaelChirico · 2019-09-20T11:41:22Z

has zlib always been a dependency for R? I'm not sure how to check

philippechataignon · 2019-09-20T14:33:32Z

Here for R 3.3.0 (may 2016) : The previously included versions of zlib, bzip2, xz and PCRE have been removed, so suitable external (usually system) versions are required.

In R 3.2.0 (april 2015): configure options --with-system-zlib, --with-system-bzlib and --with-system-pcre are now the default. For the time being there is fallback to the versions included in the R sources if no system versions are found or (unlikely) if they are too old. Linux users should check that the -devel or -dev versions of packages zlib, bzip2/libbz2 and pcre as well as xz-devel/liblzma-dev (or similar names) are installed

Commit 2001-04-06 69db71eb0c has log message "require and build zlib and libzml (not built yet)" : zlib is a dependency for R since the beginning but R source includes zlib source if there is not system zlib.

MichaelChirico · 2019-09-20T17:19:23Z

OK so noting based on the clarification from r-ver:

rocker-org/rocker-versioned#161

In order for users to be missing zlib when installing data.table, their machine has to have specifically had zlib support removed after R installation. Should be quite rare, I think the main exception being people using lean Dockerized environments such as r-ver -- and I think we're more comfortable devolving responsibility for installing to such users (especially with the note in the NEWS and we should add to Installation wiki as well).

st-pasha · 2019-09-20T18:24:21Z

For the sake of clarity, when people speak of "zlib" library, they could mean 3 different things:

zlib so/dll files, allowing zlib to be dynamically linked during the runtime. I believe that this is the version of zlib that is most commonly present on a user's system, and it is also the one that R binary depends on.
zlib a/obj file, allowing zlib to be statically linked. This would only be needed if datatable wanted to statically link with that library.
zlib h files, allowing compiling of the code that uses zlib. These files may or may not be present on user's system, and I don't know whether R gives any promises that they will ship these header files together with R's installation. However I can see that on my Mac's R 3.5.2.2 these headers are not included.

MichaelChirico · 2019-09-20T18:38:44Z

I don't know whether R gives any promises that they will ship these header files together with R's installation.

I think this is covered by Philippe's first note from R NEWS:

https://github.com/wch/r-source/blob/1a74f78a3c9e751f465b6eb27f7c1a8bf5ff2ada/doc/NEWS.Rd#L4056-L4059

philippechataignon · 2019-09-20T18:48:15Z

NEWS says that zlib headers are required to compile R but when you install R from a binary package (very usual) there no need to have zlib headers present.

vikram-rawat · 2019-10-10T17:19:47Z

R-base-dev brings some tools with it so to ensure you can compile basic programmes.

And it also pulls in zlib1g-dev as on Debian/Ubuntu systems

So just install r-base-dev along with r-base-core should be enough

The 'zlib1g-dev' missing is an indication of r-base-dev not installed, and the first step (on Debian and Ubuntu) before building packages from source should be to haev r-base-dev.

Please mention this step in news section on wiki page

mattdowle · 2019-10-13T18:03:37Z

@vikram-rawat data.table has a configure now (v1.12.5+) which points the user to install zlib1g-dev directly. See https://github.com/Rdatatable/data.table/blob/master/configure.

  pkg-config did not detect zlib is installed. Try installing:
  * deb: zlib1g-dev (Debian, Ubuntu, ...)
  * rpm: zlib-devel (Fedora, EPEL, ...)
  * brew: zlib (OSX)

For others finding this in future, only those installing from source need this. Windows and MacOS users who install the CRAN binary do not compile the package so they don't need this.

MichaelChirico added this to the 1.12.4 milestone Sep 15, 2019

jangorecki added the dev label Sep 17, 2019

jangorecki changed the title ~~Build could offer instructions for installing zlib~~ Handling of zlib fwrite dependency Sep 18, 2019

jangorecki mentioned this issue Sep 19, 2019

add zlib info to NEWS #3897

Merged

mattdowle modified the milestones: 1.12.4, 1.13.0 Sep 19, 2019

MichaelChirico mentioned this issue Sep 20, 2019

Clarification re:zlib rocker-org/rocker-versioned#161

Open

mattdowle modified the milestones: 1.12.7, 1.12.5 Oct 9, 2019

mattdowle mentioned this issue Oct 9, 2019

fwrite zlib #3951

Merged

mattdowle closed this as completed in #3951 Oct 9, 2019

jangorecki mentioned this issue Dec 12, 2020

makes dependency on zlib optional #4844

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handling of zlib fwrite dependency #3872

Handling of zlib fwrite dependency #3872

MichaelChirico commented Sep 14, 2019

MichaelChirico commented Sep 14, 2019

jangorecki commented Sep 14, 2019

st-pasha commented Sep 14, 2019

jangorecki commented Sep 15, 2019 •

edited

MichaelChirico commented Sep 16, 2019 •

edited by jangorecki

philippechataignon commented Sep 16, 2019 •

edited

philippechataignon commented Sep 16, 2019

jangorecki commented Sep 16, 2019

philippechataignon commented Sep 19, 2019

MichaelChirico commented Sep 19, 2019 •

edited

philippechataignon commented Sep 19, 2019

mattdowle commented Sep 19, 2019 •

edited

jangorecki commented Sep 20, 2019 •

edited

MichaelChirico commented Sep 20, 2019

philippechataignon commented Sep 20, 2019

MichaelChirico commented Sep 20, 2019

st-pasha commented Sep 20, 2019

MichaelChirico commented Sep 20, 2019

philippechataignon commented Sep 20, 2019

vikram-rawat commented Oct 10, 2019

mattdowle commented Oct 13, 2019 •

edited

Handling of zlib fwrite dependency #3872

Handling of zlib fwrite dependency #3872

Comments

MichaelChirico commented Sep 14, 2019

MichaelChirico commented Sep 14, 2019

jangorecki commented Sep 14, 2019

st-pasha commented Sep 14, 2019

jangorecki commented Sep 15, 2019 • edited

MichaelChirico commented Sep 16, 2019 • edited by jangorecki

philippechataignon commented Sep 16, 2019 • edited

philippechataignon commented Sep 16, 2019

jangorecki commented Sep 16, 2019

philippechataignon commented Sep 19, 2019

MichaelChirico commented Sep 19, 2019 • edited

philippechataignon commented Sep 19, 2019

mattdowle commented Sep 19, 2019 • edited

jangorecki commented Sep 20, 2019 • edited

MichaelChirico commented Sep 20, 2019

philippechataignon commented Sep 20, 2019

MichaelChirico commented Sep 20, 2019

st-pasha commented Sep 20, 2019

MichaelChirico commented Sep 20, 2019

philippechataignon commented Sep 20, 2019

vikram-rawat commented Oct 10, 2019

mattdowle commented Oct 13, 2019 • edited

jangorecki commented Sep 15, 2019 •

edited

MichaelChirico commented Sep 16, 2019 •

edited by jangorecki

philippechataignon commented Sep 16, 2019 •

edited

MichaelChirico commented Sep 19, 2019 •

edited

mattdowle commented Sep 19, 2019 •

edited

jangorecki commented Sep 20, 2019 •

edited

mattdowle commented Oct 13, 2019 •

edited