Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Fixed the leak responsible for the accumulation of unreferenced scalars #1

Merged
merged 13 commits into from

2 participants

Stanislaw Pusep Olivier Mengué
Stanislaw Pusep
Collaborator

The leak occurs due to creation of empty base object inside the XS code. This fix also happens to solve the "Attempt to free unreferenced scalar: SV 0xdeadbeef during global destruction." issue.
Modifications:

  1. Reimplemented t/old-13slowleak.t as t/96-leak.t, using Devel::Leak. It now runs 10_000 iterations instead of 200. However, it's still fast, as the downloaded URL is "file:///$Bin/$Script" (by default). This was required to spot the problem.
  2. Updated Makefile.PL so pkg-config is now able to detect libcurl in non-standard locations (like $HOME). This was required for me to deploy test cases on multiple perlbrew-based installations.
  3. Updated inc/symbols-in-versions from libcurl/7.28.0 and created a .gitignore file. Not actually required to fix the leak, but hell, why not? :)
  4. The fix itself: in Curl.xs, HASHREF_BY_DEFAULT definition was updated, reverting the order of newRV_noinc() / sv_2mortal().

The dismissal of *"Attempt to free unreferenced scalar..." warning can be verified via Net::Curl::Simple test suite; while using the original Net::Curl, it repeatedly produces the aforementioned message:

...
t/42-coro-anyevent.t ..... 1/20 Please rebuild libcurl with AsynchDNS to avoid blocking DNS requests
# loaded implementation: Net/Curl/Simple/Async/AnyEvent.pm
Attempt to free unreferenced scalar: SV 0x7fa88b964378 during global destruction.
t/42-coro-anyevent.t ..... ok
...

I also "smoked" the patched Net::Curl on a reasonably large crawler system (which processes over 1 million HTML-pages every 24 hours) no unexpected behavior was detected.

Olivier Mengué

Reverse link to rt.cpan.org: RT#81053

Stanislaw Pusep creaktive merged commit ccddffe into from
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
16 .gitignore
View
@@ -0,0 +1,16 @@
+Curl.bs
+Curl.[co]
+MYMETA.*
+Makefile
+blib/*
+*.inc
+lib/Net/Curl/Compat.pm
+lib/Net/Curl/examples.pod
+pm_to_blib
+Net-Curl-*/*
+*.gz
+*.bak
+*.old
+*~
+nytprof*
+cover_db/*
5 Curl.xs
View
@@ -376,7 +376,10 @@ perl_curl_constant_add( pTHX_ HV *hash, const char *name, I32 namelen,
if ( SvOK( *sv ) || SvTYPE( *sv ) == SVt_PVGV ) {
newCONSTSUB( hash, name, value );
} else {
+#pragma clang diagnostic push
+#pragma clang diagnostic ignored "-Wunused-value"
SvUPGRADE( *sv, SVt_RV );
+#pragma clang diagnostic pop
SvRV_set( *sv, value );
SvROK_on( *sv );
SvREADONLY_on( value );
@@ -409,7 +412,7 @@ typedef perl_curl_multi_t *Net__Curl__Multi;
typedef perl_curl_share_t *Net__Curl__Share;
/* default base object */
-#define HASHREF_BY_DEFAULT newRV_noinc( sv_2mortal( (SV *) newHV() ) )
+#define HASHREF_BY_DEFAULT sv_2mortal( newRV_noinc( (SV *) newHV() ) )
#include "curl-Easy-c.inc"
#include "curl-Form-c.inc"
2  Curl_Easy.xsh
View
@@ -571,8 +571,6 @@ pushopt( easy, option, value )
CURLcode ret;
CODE:
ret = perl_curl_easy_setoptslist( aTHX_ easy, option, value, 0 );
- if ( ret < 0 )
- ret = CURLE_BAD_FUNCTION_ARGUMENT;
EASY_DIE( ret );
2  MANIFEST
View
@@ -38,6 +38,7 @@ t/51-crash-destroy-with-callbacks.t
t/52-alter-base.t
t/53-crash-destroy-with-callbacks-multi.t
t/54-crash-getinfo-slist.t
+t/96-leak.t
t/97-pod.t
t/98-pod-coverage.t
t/99-symbols.t
@@ -65,7 +66,6 @@ t/old-07ftp-upload.t
t/old-08ssl.t
t/old-09times.t
t/old-10errbuf.t
-t/old-13slowleak.t
t/old-14duphandle.t
t/old-15duphandle-callback.t
t/old-16formpost.t
8 Makefile.PL
View
@@ -35,6 +35,14 @@ if ( $curl{libs} and $curl{version} ) {
} else {
eval {
require ExtUtils::PkgConfig;
+ my $pkgconfig = `which curl`;
+ unless ( $? ) {
+ chomp $pkgconfig;
+ $pkgconfig =~ s/\bbin(.+?)curl$/lib\1pkgconfig/i;
+ $ENV{PKG_CONFIG_PATH} = $pkgconfig;
+ } else {
+ print STDERR "which failed:\n$@\n";
+ }
%curl = ExtUtils::PkgConfig->find( 'libcurl' );
$curl{version} = $curl{modversion};
};
36 inc/symbols-in-versions
View
@@ -20,6 +20,7 @@ CURLAUTH_DIGEST_IE 7.19.3
CURLAUTH_GSSNEGOTIATE 7.10.6
CURLAUTH_NONE 7.10.6
CURLAUTH_NTLM 7.10.6
+CURLAUTH_NTLM_WB 7.22.0
CURLAUTH_ONLY 7.21.3
CURLCLOSEPOLICY_CALLBACK 7.7
CURLCLOSEPOLICY_LEAST_RECENTLY_USED 7.7
@@ -44,6 +45,8 @@ CURLE_COULDNT_RESOLVE_PROXY 7.1
CURLE_FAILED_INIT 7.1
CURLE_FILESIZE_EXCEEDED 7.10.8
CURLE_FILE_COULDNT_READ_FILE 7.1
+CURLE_FTP_ACCEPT_FAILED 7.24.0
+CURLE_FTP_ACCEPT_TIMEOUT 7.24.0
CURLE_FTP_ACCESS_DENIED 7.1
CURLE_FTP_BAD_DOWNLOAD_RESUME 7.1 7.1
CURLE_FTP_BAD_FILE_LIST 7.21.0
@@ -82,6 +85,7 @@ CURLE_LDAP_SEARCH_FAILED 7.1
CURLE_LIBRARY_NOT_FOUND 7.1 7.17.0
CURLE_LOGIN_DENIED 7.13.1
CURLE_MALFORMAT_USER 7.1 7.17.0
+CURLE_NO_CONNECTION_AVAILABLE 7.30.0
CURLE_NOT_BUILT_IN 7.21.5
CURLE_OK 7.1
CURLE_OPERATION_TIMEDOUT 7.10.2
@@ -186,6 +190,9 @@ CURLFTPSSL_TRY 7.11.0 7.17.0
CURLFTP_CREATE_DIR 7.19.4
CURLFTP_CREATE_DIR_NONE 7.19.4
CURLFTP_CREATE_DIR_RETRY 7.19.4
+CURLGSSAPI_DELEGATION_FLAG 7.22.0
+CURLGSSAPI_DELEGATION_NONE 7.22.0
+CURLGSSAPI_DELEGATION_POLICY_FLAG 7.22.0
CURLINFO_APPCONNECT_TIME 7.19.0
CURLINFO_CERTINFO 7.19.1
CURLINFO_CONDITION_UNMET 7.19.4
@@ -261,8 +268,15 @@ CURLKHTYPE_DSS 7.19.6
CURLKHTYPE_RSA 7.19.6
CURLKHTYPE_RSA1 7.19.6
CURLKHTYPE_UNKNOWN 7.19.6
+CURLMOPT_CHUNK_LENGTH_PENALTY_SIZE 7.30.0
+CURLMOPT_CONTENT_LENGTH_PENALTY_SIZE 7.30.0
+CURLMOPT_MAX_HOST_CONNECTIONS 7.30.0
+CURLMOPT_MAX_PIPELINE_LENGTH 7.30.0
+CURLMOPT_MAX_TOTAL_CONNECTIONS 7.30.0
CURLMOPT_MAXCONNECTS 7.16.3
CURLMOPT_PIPELINING 7.16.0
+CURLMOPT_PIPELINING_SERVER_BL 7.30.0
+CURLMOPT_PIPELINING_SITE_BL 7.30.0
CURLMOPT_SOCKETDATA 7.15.4
CURLMOPT_SOCKETFUNCTION 7.15.4
CURLMOPT_TIMERDATA 7.16.0
@@ -282,6 +296,7 @@ CURLOPTTYPE_FUNCTIONPOINT 7.1
CURLOPTTYPE_LONG 7.1
CURLOPTTYPE_OBJECTPOINT 7.1
CURLOPTTYPE_OFF_T 7.11.0
+CURLOPT_ACCEPTTIMEOUT_MS 7.24.0
CURLOPT_ACCEPT_ENCODING 7.21.6
CURLOPT_ADDRESS_SCOPE 7.19.0
CURLOPT_APPEND 7.17.0
@@ -316,6 +331,7 @@ CURLOPT_DEBUGDATA 7.9.6
CURLOPT_DEBUGFUNCTION 7.9.6
CURLOPT_DIRLISTONLY 7.17.0
CURLOPT_DNS_CACHE_TIMEOUT 7.9.3
+CURLOPT_DNS_SERVERS 7.24.0
CURLOPT_DNS_USE_GLOBAL_CACHE 7.9.3 7.11.1
CURLOPT_EGDSOCKET 7.7
CURLOPT_ENCODING 7.10
@@ -344,6 +360,7 @@ CURLOPT_FTP_SSL_CCC 7.16.1
CURLOPT_FTP_USE_EPRT 7.10.5
CURLOPT_FTP_USE_EPSV 7.9.2
CURLOPT_FTP_USE_PRET 7.20.0
+CURLOPT_GSSAPI_DELEGATION 7.22.0
CURLOPT_HEADER 7.1
CURLOPT_HEADERDATA 7.10
CURLOPT_HEADERFUNCTION 7.7.2
@@ -375,6 +392,7 @@ CURLOPT_LOCALPORT 7.15.2
CURLOPT_LOCALPORTRANGE 7.15.2
CURLOPT_LOW_SPEED_LIMIT 7.1
CURLOPT_LOW_SPEED_TIME 7.1
+CURLOPT_MAIL_AUTH 7.25.0
CURLOPT_MAIL_FROM 7.20.0
CURLOPT_MAIL_RCPT 7.20.0
CURLOPT_MAXCONNECTS 7.7
@@ -398,7 +416,7 @@ CURLOPT_OPENSOCKETFUNCTION 7.17.1
CURLOPT_PASSWDDATA 7.4.2 7.11.1 7.15.5
CURLOPT_PASSWDFUNCTION 7.4.2 7.11.1 7.15.5
CURLOPT_PASSWORD 7.19.1
-CURLOPT_PASV_HOST 7.12.1 7.15.6 7.15.5
+CURLOPT_PASV_HOST 7.12.1 7.16.0 7.15.5
CURLOPT_PORT 7.1
CURLOPT_POST 7.1
CURLOPT_POST301 7.17.1 7.19.1
@@ -473,10 +491,14 @@ CURLOPT_SSLVERSION 7.1
CURLOPT_SSL_CIPHER_LIST 7.9
CURLOPT_SSL_CTX_DATA 7.10.6
CURLOPT_SSL_CTX_FUNCTION 7.10.6
+CURLOPT_SSL_OPTIONS 7.25.0
CURLOPT_SSL_SESSIONID_CACHE 7.16.0
CURLOPT_SSL_VERIFYHOST 7.8.1
CURLOPT_SSL_VERIFYPEER 7.4.2
CURLOPT_STDERR 7.1
+CURLOPT_TCP_KEEPALIVE 7.25.0
+CURLOPT_TCP_KEEPIDLE 7.25.0
+CURLOPT_TCP_KEEPINTVL 7.25.0
CURLOPT_TCP_NODELAY 7.11.2
CURLOPT_TELNETOPTIONS 7.7
CURLOPT_TFTP_BLKSIZE 7.19.4
@@ -545,6 +567,7 @@ CURLSHE_BAD_OPTION 7.10.3
CURLSHE_INVALID 7.10.3
CURLSHE_IN_USE 7.10.3
CURLSHE_NOMEM 7.12.0
+CURLSHE_NOT_BUILT_IN 7.23.0
CURLSHE_OK 7.10.3
CURLSHOPT_LOCKFUNC 7.10.3
CURLSHOPT_NONE 7.10.3
@@ -552,7 +575,9 @@ CURLSHOPT_SHARE 7.10.3
CURLSHOPT_UNLOCKFUNC 7.10.3
CURLSHOPT_UNSHARE 7.10.3
CURLSHOPT_USERDATA 7.10.3
-CURLSOCKTYPE_IPCXN 7.15.6
+CURLSOCKTYPE_ACCEPT 7.28.0
+CURLSOCKTYPE_IPCXN 7.16.0
+CURLSSH_AUTH_AGENT 7.28.0
CURLSSH_AUTH_ANY 7.16.1
CURLSSH_AUTH_DEFAULT 7.16.1
CURLSSH_AUTH_HOST 7.16.1
@@ -560,6 +585,7 @@ CURLSSH_AUTH_KEYBOARD 7.16.1
CURLSSH_AUTH_NONE 7.16.1
CURLSSH_AUTH_PASSWORD 7.16.1
CURLSSH_AUTH_PUBLICKEY 7.16.1
+CURLSSLOPT_ALLOW_BEAST 7.25.0
CURLUSESSL_ALL 7.17.0
CURLUSESSL_CONTROL 7.17.0
CURLUSESSL_NONE 7.17.0
@@ -596,6 +622,7 @@ CURL_GLOBAL_DEFAULT 7.8
CURL_GLOBAL_NOTHING 7.8
CURL_GLOBAL_SSL 7.8
CURL_GLOBAL_WIN32 7.8.1
+CURL_GLOBAL_ACK_EINTR 7.30.0
CURL_HTTP_VERSION_1_0 7.9.1
CURL_HTTP_VERSION_1_1 7.9.1
CURL_HTTP_VERSION_NONE 7.9.1
@@ -633,6 +660,7 @@ CURL_READFUNC_PAUSE 7.18.0
CURL_REDIR_GET_ALL 7.19.1
CURL_REDIR_POST_301 7.19.1
CURL_REDIR_POST_302 7.19.1
+CURL_REDIR_POST_303 7.25.1
CURL_REDIR_POST_ALL 7.19.1
CURL_RTSPREQ_ANNOUNCE 7.20.0
CURL_RTSPREQ_DESCRIBE 7.20.0
@@ -675,8 +703,12 @@ CURL_VERSION_KERBEROS4 7.10
CURL_VERSION_LARGEFILE 7.11.1
CURL_VERSION_LIBZ 7.10
CURL_VERSION_NTLM 7.10.6
+CURL_VERSION_NTLM_WB 7.22.0
CURL_VERSION_SPNEGO 7.10.8
CURL_VERSION_SSL 7.10
CURL_VERSION_SSPI 7.13.2
CURL_VERSION_TLSAUTH_SRP 7.21.4
+CURL_WAIT_POLLIN 7.28.0
+CURL_WAIT_POLLOUT 7.28.0
+CURL_WAIT_POLLPRI 7.28.0
CURL_WRITEFUNC_PAUSE 7.18.0
92 t/96-leak.t
View
@@ -0,0 +1,92 @@
+#!/usr/bin/perl
+use strict;
+use warnings qw(all);
+use FindBin qw($Bin $Script);
+
+# shamelessly stolen from RJRAY/Perl-RPM-1.51/t/09_leaks.t
+
+BEGIN {
+ eval {
+ require Devel::Leak;
+ require Test::More;
+ };
+ if ($@) {
+ print "1..0 # Skip Devel::Leak and Test::More required\n";
+ exit 0;
+ }
+}
+
+use Test::More;
+
+sub test_leak (&$;$) {
+ my ($code, $descr, $maxleak) = (@_, 0);
+ my $n1 = Devel::Leak::NoteSV(my $handle);
+ $code->() for 1 .. 10_000;
+ my $n2 = Devel::Leak::CheckSV($handle);
+ cmp_ok($n1 + $maxleak, '>=', $n2, $descr);
+}
+
+use Net::Curl qw(:constants);
+use Net::Curl::Easy qw(:constants);
+use Net::Curl::Form qw(:constants);
+use Net::Curl::Multi qw(:constants);
+use Net::Curl::Share qw(:constants);
+
+my $easy = Net::Curl::Easy->new;
+test_leak { my $easy = Net::Curl::Easy->new or die }
+ q(Net::Curl::Easy->new);
+
+my $form = Net::Curl::Form->new;
+test_leak { my $form = Net::Curl::Form->new or die }
+ q(Net::Curl::Form->new);
+
+my $multi = Net::Curl::Multi->new;
+SKIP: {
+ skip q(libcurl/7.29.0 crashes here: http://sourceforge.net/p/curl/bugs/1194/), 1
+ if Net::Curl::version_info()->{version} eq q(7.29.0);
+ test_leak { my $multi = Net::Curl::Multi->new or die }
+ q(Net::Curl::Multi->new);
+}
+
+my $share = Net::Curl::Share->new;
+$share->setopt(CURLSHOPT_SHARE, CURL_LOCK_DATA_COOKIE);
+$share->setopt(CURLSHOPT_SHARE, CURL_LOCK_DATA_DNS);
+eval { $share->setopt(CURLSHOPT_SHARE, CURL_LOCK_DATA_SSL_SESSION) };
+test_leak { my $share = Net::Curl::Share->new or die }
+ q(Net::Curl::Share->new);
+
+my $url = $ENV{CURL_TEST_URL};
+$url = qq(file://$Bin/$Script)
+ if not defined $url or $url;
+
+my $n1 = Devel::Leak::NoteSV(my $handle);
+test_leak {
+ my $curl = Net::Curl::Easy->new() or die "cannot curl";
+ $multi->add_handle($curl);
+
+ $curl->setopt(CURLOPT_NOPROGRESS, 1);
+ $curl->setopt(CURLOPT_FOLLOWLOCATION, 1);
+ $curl->setopt(CURLOPT_TIMEOUT, 30);
+
+ open(my $head, "+>", undef);
+ Net::Curl::Easy::setopt($curl, CURLOPT_WRITEHEADER, $head);
+ open(my $body, "+>", undef);
+ Net::Curl::Easy::setopt($curl, CURLOPT_WRITEDATA, $body);
+
+ $curl->setopt(CURLOPT_URL, $url);
+ $curl->setopt(CURLOPT_SHARE, $share);
+
+ eval { $multi->perform() };
+ if (not $@) {
+ my $bytes = $curl->getinfo(CURLINFO_SIZE_DOWNLOAD);
+ my $realurl = $curl->getinfo(CURLINFO_EFFECTIVE_URL);
+ my $httpcode = $curl->getinfo(CURLINFO_HTTP_CODE);
+ $multi->remove_handle($curl);
+ } else {
+ die "not ok " . $curl->error;
+ }
+} q(old-13slowleak.t);
+my $n2 = Devel::Leak::CheckSV($handle);
+cmp_ok($n1, '>=', $n2, q(cross-references));
+
+done_testing(6);
43 t/old-13slowleak.t
View
@@ -1,43 +0,0 @@
-#!perl
-
-use strict;
-use warnings;
-#use Test::More tests => 214;
-use Test::More skip_all => "Not performing slow leakage regression test";
-
-BEGIN { use_ok( 'Net::Curl::Easy' ); }
-use Net::Curl::Easy qw(:constants);
-
-my $url = $ENV{CURL_TEST_URL};
-
-# There was a slow leak per curl handle init/cleanup. Hopefully fixed.
-
-foreach my $j (1..200) {
-
-# Init the curl session
-my $curl = Net::Curl::Easy->new() or die "cannot curl";
-
-$curl->setopt(CURLOPT_NOPROGRESS, 1);
-$curl->setopt(CURLOPT_FOLLOWLOCATION, 1);
-$curl->setopt(CURLOPT_TIMEOUT, 30);
-
-open (HEAD, "+>",undef);
-Net::Curl::Easy::setopt($curl, CURLOPT_WRITEHEADER, \*HEAD);
-open (BODY, "+>", undef);
-Net::Curl::Easy::setopt($curl, CURLOPT_FILE, \*BODY);
-
-$curl->setopt(CURLOPT_URL, $url);
-
-my $httpcode = 0;
-
-eval { $curl->perform(); };
-if ( not $@ ) {
- my bytes=$curl->getinfo(CURLINFO_SIZE_DOWNLOAD);
- my $realurl=$curl->getinfo(CURLINFO_EFFECTIVE_URL);
- my $httpcode=$curl->getinfo(CURLINFO_HTTP_CODE);
-} else {
- print "not ok $retcode / ".$curl->error."\n";
-}
-
-}
-
Something went wrong with that request. Please try again.