Error when compiling with clang with -std=c++11 -stdlib=libc++

[vigsterkr]

When trying to use c++11 mode with clang, i.e. the -std=c++11 -stdlib=libc++ flags are set, the swig generated cxx cannot be compiled.

An example of the error:

/Users/wiking/shogun/build/src/interfaces/ruby_modular/modshogunRUBY_wrap.cxx:11297:24: error: no matching member function for call to 'erase'
      result = (arg1)->erase(arg2);
               ~~~~~~~~^~~~~
/usr/bin/../lib/c++/v1/vector:716:76: note: candidate function not viable: no known conversion from 'SwigValueWrapper<std::vector<int>::iterator>' to 'const_iterator' (aka '__wrap_iter<const_pointer>') for 1st argument
    __attribute__ ((__visibility__("hidden"), __always_inline__)) iterator erase(const_iterator __position);
                                                                           ^
/usr/bin/../lib/c++/v1/vector:717:14: note: candidate function not viable: requires 2 arguments, but 1 was provided
    iterator erase(const_iterator __first, const_iterator __last);
             ^
/Users/wiking/shogun/build/src/interfaces/ruby_modular/modshogunRUBY_wrap.cxx:11376:24: error: no matching member function for call to 'erase'
      result = (arg1)->erase(arg2,arg3);
               ~~~~~~~~^~~~~
/usr/bin/../lib/c++/v1/vector:717:14: note: candidate function not viable: no known conversion from 'SwigValueWrapper<std::vector<int>::iterator>' to 'const_iterator' (aka '__wrap_iter<const_pointer>') for 1st argument
    iterator erase(const_iterator __first, const_iterator __last);
             ^
/usr/bin/../lib/c++/v1/vector:716:76: note: candidate function not viable: requires single argument '__position', but 2 arguments were provided
    __attribute__ ((__visibility__("hidden"), __always_inline__)) iterator erase(const_iterator __position);
                                                                           ^
/Users/wiking/shogun/build/src/interfaces/ruby_modular/modshogunRUBY_wrap.cxx:11982:24: error: no matching member function for call to 'insert'
      result = (arg1)->insert(arg2,(std::vector< int >::value_type const &)*arg3);
               ~~~~~~~~^~~~~~
/usr/bin/../lib/c++/v1/vector:686:14: note: candidate function not viable: no known conversion from 'SwigValueWrapper<std::vector<int>::iterator>' to 'const_iterator' (aka '__wrap_iter<const_pointer>') for 1st argument
    iterator insert(const_iterator __position, const_reference __x);
             ^
/usr/bin/../lib/c++/v1/vector:688:14: note: candidate function not viable: no known conversion from 'SwigValueWrapper<std::vector<int>::iterator>' to 'const_iterator' (aka '__wrap_iter<const_pointer>') for 1st argument
    iterator insert(const_iterator __position, value_type&& __x);
             ^
/usr/bin/../lib/c++/v1/vector:712:14: note: candidate function not viable: no known conversion from 'SwigValueWrapper<std::vector<int>::iterator>' to 'const_iterator' (aka '__wrap_iter<const_pointer>') for 1st argument
    iterator insert(const_iterator __position, initializer_list<value_type> __il)
             ^
/usr/bin/../lib/c++/v1/vector:702:9: note: candidate function template not viable: requires 3 arguments, but 2 were provided
        insert(const_iterator __position, _InputIterator __first, _InputIterator __last);
        ^
/usr/bin/../lib/c++/v1/vector:709:9: note: candidate function template not viable: requires 3 arguments, but 2 were provided
        insert(const_iterator __position, _ForwardIterator __first, _ForwardIterator __last);
        ^
/usr/bin/../lib/c++/v1/vector:694:14: note: candidate function not viable: requires 3 arguments, but 2 were provided
    iterator insert(const_iterator __position, size_type __n, const_reference __x);
             ^
/Users/wiking/shogun/build/src/interfaces/ruby_modular/modshogunRUBY_wrap.cxx:12063:15: error: no matching member function for call to 'insert'
      (arg1)->insert(arg2,arg3,(std::vector< int >::value_type const &)*arg4);
      ~~~~~~~~^~~~~~
/usr/bin/../lib/c++/v1/vector:694:14: note: candidate function not viable: no known conversion from 'SwigValueWrapper<std::vector<int>::iterator>' to 'const_iterator' (aka '__wrap_iter<const_pointer>') for 1st argument
    iterator insert(const_iterator __position, size_type __n, const_reference __x);
             ^
/usr/bin/../lib/c++/v1/vector:702:9: note: candidate template ignored: deduced conflicting types for parameter '_InputIterator' ('unsigned long' vs. 'int')
        insert(const_iterator __position, _InputIterator __first, _InputIterator __last);
        ^
/usr/bin/../lib/c++/v1/vector:709:9: note: candidate template ignored: deduced conflicting types for parameter '_ForwardIterator' ('unsigned long' vs. 'int')
        insert(const_iterator __position, _ForwardIterator __first, _ForwardIterator __last);
        ^
/usr/bin/../lib/c++/v1/vector:686:14: note: candidate function not viable: requires 2 arguments, but 3 were provided
    iterator insert(const_iterator __position, const_reference __x);
             ^
/usr/bin/../lib/c++/v1/vector:688:14: note: candidate function not viable: requires 2 arguments, but 3 were provided
    iterator insert(const_iterator __position, value_type&& __x);
             ^
/usr/bin/../lib/c++/v1/vector:712:14: note: candidate function not viable: requires 2 arguments, but 3 were provided
    iterator insert(const_iterator __position, initializer_list<value_type> __il)
             ^
/Users/wiking/shogun/build/src/interfaces/ruby_modular/modshogunRUBY_wrap.cxx:14674:24: error: no matching member function for call to 'erase'
      result = (arg1)->erase(arg2);
               ~~~~~~~~^~~~~
/usr/bin/../lib/c++/v1/vector:716:76: note: candidate function not viable: no known conversion from 'SwigValueWrapper<std::vector<double>::iterator>' to 'const_iterator' (aka '__wrap_iter<const_pointer>') for 1st argument
    __attribute__ ((__visibility__("hidden"), __always_inline__)) iterator erase(const_iterator __position);
                                                                           ^
/usr/bin/../lib/c++/v1/vector:717:14: note: candidate function not viable: requires 2 arguments, but 1 was provided
    iterator erase(const_iterator __first, const_iterator __last);
             ^
/Users/wiking/shogun/build/src/interfaces/ruby_modular/modshogunRUBY_wrap.cxx:14753:24: error: no matching member function for call to 'erase'
      result = (arg1)->erase(arg2,arg3);
               ~~~~~~~~^~~~~
/usr/bin/../lib/c++/v1/vector:717:14: note: candidate function not viable: no known conversion from 'SwigValueWrapper<std::vector<double>::iterator>' to 'const_iterator' (aka '__wrap_iter<const_pointer>') for 1st argument
    iterator erase(const_iterator __first, const_iterator __last);
             ^
/usr/bin/../lib/c++/v1/vector:716:76: note: candidate function not viable: requires single argument '__position', but 2 arguments were provided
    __attribute__ ((__visibility__("hidden"), __always_inline__)) iterator erase(const_iterator __position);
                                                                           ^
/Users/wiking/shogun/build/src/interfaces/ruby_modular/modshogunRUBY_wrap.cxx:15359:24: error: no matching member function for call to 'insert'
      result = (arg1)->insert(arg2,(std::vector< double >::value_type const &)*arg3);
               ~~~~~~~~^~~~~~
/usr/bin/../lib/c++/v1/vector:686:14: note: candidate function not viable: no known conversion from 'SwigValueWrapper<std::vector<double>::iterator>' to 'const_iterator' (aka '__wrap_iter<const_pointer>') for 1st argument
    iterator insert(const_iterator __position, const_reference __x);
             ^
/usr/bin/../lib/c++/v1/vector:688:14: note: candidate function not viable: no known conversion from 'SwigValueWrapper<std::vector<double>::iterator>' to 'const_iterator' (aka '__wrap_iter<const_pointer>') for 1st argument
    iterator insert(const_iterator __position, value_type&& __x);
             ^
/usr/bin/../lib/c++/v1/vector:712:14: note: candidate function not viable: no known conversion from 'SwigValueWrapper<std::vector<double>::iterator>' to 'const_iterator' (aka '__wrap_iter<const_pointer>') for 1st argument
    iterator insert(const_iterator __position, initializer_list<value_type> __il)
             ^
/usr/bin/../lib/c++/v1/vector:702:9: note: candidate function template not viable: requires 3 arguments, but 2 were provided
        insert(const_iterator __position, _InputIterator __first, _InputIterator __last);
        ^
/usr/bin/../lib/c++/v1/vector:709:9: note: candidate function template not viable: requires 3 arguments, but 2 were provided
        insert(const_iterator __position, _ForwardIterator __first, _ForwardIterator __last);
        ^
/usr/bin/../lib/c++/v1/vector:694:14: note: candidate function not viable: requires 3 arguments, but 2 were provided
    iterator insert(const_iterator __position, size_type __n, const_reference __x);
             ^
/Users/wiking/shogun/build/src/interfaces/ruby_modular/modshogunRUBY_wrap.cxx:15440:15: error: no matching member function for call to 'insert'
      (arg1)->insert(arg2,arg3,(std::vector< double >::value_type const &)*arg4);
      ~~~~~~~~^~~~~~
/usr/bin/../lib/c++/v1/vector:694:14: note: candidate function not viable: no known conversion from 'SwigValueWrapper<std::vector<double>::iterator>' to 'const_iterator' (aka '__wrap_iter<const_pointer>') for 1st argument
    iterator insert(const_iterator __position, size_type __n, const_reference __x);
             ^
/usr/bin/../lib/c++/v1/vector:702:9: note: candidate template ignored: deduced conflicting types for parameter '_InputIterator' ('unsigned long' vs. 'double')
        insert(const_iterator __position, _InputIterator __first, _InputIterator __last);
        ^
/usr/bin/../lib/c++/v1/vector:709:9: note: candidate template ignored: deduced conflicting types for parameter '_ForwardIterator' ('unsigned long' vs. 'double')
        insert(const_iterator __position, _ForwardIterator __first, _ForwardIterator __last);
        ^
/usr/bin/../lib/c++/v1/vector:686:14: note: candidate function not viable: requires 2 arguments, but 3 were provided
    iterator insert(const_iterator __position, const_reference __x);
             ^
/usr/bin/../lib/c++/v1/vector:688:14: note: candidate function not viable: requires 2 arguments, but 3 were provided
    iterator insert(const_iterator __position, value_type&& __x);
             ^
/usr/bin/../lib/c++/v1/vector:712:14: note: candidate function not viable: requires 2 arguments, but 3 were provided
    iterator insert(const_iterator __position, initializer_list<value_type> __il)
             ^
8 errors generated.

[m7thon]

In c++11, the std::vector methods erase and insert take const_iterators instead of iterators, and for some reason automatic conversion does not work. In any case, one way to fix this is to change this accordingly in the file std_container.i.

I also needed to compile with -Wno-c++11-narrowing to avoid errors of the kind:

swig_python/tom_wrap.cxx:24970:5: error: non-constant-expression cannot be narrowed from type 'size_t' (aka 'unsigned long') to 'Py_ssize_t' (aka 'long') in initializer list [-Wc++11-narrowing]
    (size_t)(((char*)&((SwigPyObject *) 64L)->dict) - (char*) 64L), /* tp_dictoffset */
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
swig_python/tom_wrap.cxx:24970:5: note: override this message by inserting an explicit cast
    (size_t)(((char*)&((SwigPyObject *) 64L)->dict) - (char*) 64L), /* tp_dictoffset */
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    static_cast<Py_ssize_t>(                                      )

While easy to fix, I couldn't find where the latter code snippet is generated from.

[ojwb]

I changed that rather icky code to simply use the standard offsetof macro a few months ago in b477cb6, which is in 2.0.11, so I'd suggest upgrading.

[m7thon]

Yes, thanks, I just updated to 2.0.11 and the latter error is indeed fixed. But in std/std_container.i the member functions erase and insert do need to be changed to take const_iterators for c++11, no?

[vigsterkr]

@ojwb indeed, compiling with llvm's libc++ the insert function will fail

[PaulPrice]

I've put the first steps towards a fix on my branch issues/73 at PaulPrice/swig@11908dec5b715014cecfc455e97d19582ad44db1 . This is certainly not anywhere near ready for integration, but it does allow building a SWIG wrapper with libc++.

I've seen a few segfaults (which I've not yet been able to debug) in tests, but at least it's a step forward.

[wsfulton]

This is challenging as the signature of the erase and insert methods have changed and I do not want to add some sort of c++11 command line option when running SWIG to differentiate between standards.

I have just discovered the value of __cplusplus is set to 201103L for C++11, which is excellent news because there is then a standard way to detect when running in C++11 or earlier mode.

#if defined(__cplusplus) && (__cplusplus >= 201103L)
// C++11 or later
#else
// pre C++11
#endif

Either we use this macro to handle the difference in some way. Or we might get away with some clever template metaprogramming to deal with the two different types in the changed api (iterator and const_iterator).

[wsfulton]

Looks like gcc-4.8.x has __cplusplus set to 201103L when using -std=c++11, however, erase and insert are using the C++03 api (non-const iterator). I checked gcc-4.8.2.

gcc-4.9 prerelease from september 2013 has __cplusplus set to 201103L when using -std=c++11 too, and it has the C++11 api for erase and insert.

So if we need to use the preprocessor we will have to additionally use some compiler version checking.

[PaulPrice]

Yeah, I did some tinkering earlier and found that __cplusplus isn't reliable for discovering the API of erase and insert. I think we're going to need some template trickery. I'm on vacation until the new year, but hope to look at it when I get back.

[DarthGandalf]

It fails even without -std=c++11. Using clang with libc++ is enough to produce those errors.

$ clang --version
Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn)
Target: x86_64-apple-darwin13.0.0
Thread model: posix

[wsfulton]

@DarthGandalf What does clang++ -E give you for the containers in question? Are the definitions of erase and insert using C++11 const_iterator or pre-C++11 iterator as the input types?

[vigsterkr]

@DarthGandalf it fails because in 10.9 OSX clang uses libc++ by default instead of libstdc++
a quickfix is to tell clang to use libstdc++

clang -stdlib=libstdc++ ...

[DarthGandalf]

@wsfulton const_iterator:

# 908 "/Library/Developer/CommandLineTools/usr/bin/../lib/c++/v1/map" 3
    __attribute__ ((__visibility__("hidden"), __always_inline__))
    pair<iterator, bool>
        insert(const value_type& __v) {return __tree_.__insert_unique(__v);}

    __attribute__ ((__visibility__("hidden"), __always_inline__))
    iterator
        insert(const_iterator __p, const value_type& __v)
            {return __tree_.__insert_unique(__p.__i_, __v);}

    template <class _InputIterator>
        __attribute__ ((__visibility__("hidden"), __always_inline__))
        void insert(_InputIterator __f, _InputIterator __l)
        {
            for (const_iterator __e = cend(); __f != __l; ++__f)
                insert(__e.__i_, *__f);
        }
# 933 "/Library/Developer/CommandLineTools/usr/bin/../lib/c++/v1/map" 3
    __attribute__ ((__visibility__("hidden"), __always_inline__))
    iterator erase(const_iterator __p) {return __tree_.erase(__p.__i_);}
    __attribute__ ((__visibility__("hidden"), __always_inline__))
    size_type erase(const key_type& __k)
        {return __tree_.__erase_unique(__k);}
    __attribute__ ((__visibility__("hidden"), __always_inline__))
    iterator erase(const_iterator __f, const_iterator __l)
        {return __tree_.erase(__f.__i_, __l.__i_);}

@vigsterkr With libstdc++ it works, yes

[wsfulton]

This is a minimal test replicating the problem:

%include <std_vector.i>

%inline %{
typedef float Real;
%}

%template(RealVector) std::vector<Real>;

The SWIG workarounds are either to change the %template to avoid use of typedef's ... there are various problems with %template and type lookups, so this is generally recommended atm anyway. So use:

%template(RealVector) std::vector;

Then the internals in SWIG will find std::vector::iterator and SwigValueWrapper will not be used and the code will compile.

The 2nd workaround is to provide a specialization for SwigValueWrapper, such as:

%{
template<> class SwigValueWrapper<std::vector<Real>::iterator> {
  struct SwigMovePointer {
    std::vector<Real>::iterator *ptr;
    SwigMovePointer(std::vector<Real>::iterator *p) : ptr(p) { }
    ~SwigMovePointer() { delete ptr; }
    SwigMovePointer& operator=(SwigMovePointer& rhs) { std::vector<Real>::iterator* oldptr = ptr; ptr = 0; delete oldptr; ptr = rhs.ptr; rhs.ptr = 0; return *this; }
  } pointer;
  SwigValueWrapper& operator=(const SwigValueWrapper<std::vector<Real>::iterator>& rhs);
  SwigValueWrapper(const SwigValueWrapper<std::vector<Real>::iterator>& rhs);
public:
  SwigValueWrapper() : pointer(0) { }
  SwigValueWrapper& operator=(const std::vector<Real>::iterator& t) { SwigMovePointer tmp(new std::vector<Real>::iterator(t)); pointer = tmp; return *this; }
  operator std::vector<Real>::iterator&() const { return *pointer.ptr; }
  operator std::vector<Real>::const_iterator&() const { return *(std::vector<Real>::const_iterator *)pointer.ptr; }
  std::vector<Real>::iterator *operator&() { return pointer.ptr; }
};
%}

Put this just before the %template. Note the 2nd operator overload to const_iterator in addition to iterator. Yuck!

Any suggestions for specializing this more generally for all containers? If not, I'll put in a workaround which is simply put in a wrapper around the erase and insert methods inside an %extend. This works because the %extend wrapper uses iterator, which SwigValueWrapper will convert to, then by simply passing the parameter on to erase in the container, normal iterator to const_iterator conversion works (SwigValueWrapper does not get in the way).

The ideal fix would be to not generate the SwigValueWrapper, but this is likely to require some risky changes in the type system which I am not going to tackle now. Also if users do not provide all the type information (such as the Real typedef), then SWIG won't be able to successfully look up iterator as a type and will fall back to SwigValueWrapper. On the other hand I think SwigValueWrapper should only be generated when SWIG is sure that there is no assignment operator. Currently it uses SwigValueWrapper for non-template types only if it is sure the type is non-copyable. If type information is missing, it does not use SwigValueWrapper. The behaviour for template types is different for some reason (SWIG will use SwigValueWrapper if it can't find any type information) and I don't think this is quite right.

[RobertLuptonTheGood]

I'm not a TMP expert, but I think it's hard to get at function (as opposed to template) arguments. You might be able to do something with SFINAE and an rvalue reference but clang fails to fail on &&.

What we can do now is to convert the const_iterator to an iterator. There's an implicit conversion back, so this should work for old and new compilers:

#include <vector>

int
main()
{
    std::vector<int> vec;
    std::vector<int>::const_iterator cptr = vec.begin();
    // Obtain an iterator from a const_iterator.  Should be cheap or a noop for a vector
    std::vector<int>::iterator ptr = vec.begin();
    std::advance(ptr, std::distance<std::vector<int>::const_iterator>(ptr, cptr));

    vec.erase(ptr);
}

[FlorianFranzen]

Is there a reason why this fix was not added to swig 2.0 as well? The freebsd-ports backport works quite well for me.

[koobs]

A backport to the 2.0 branch would be handy, its highly desirable to only keep local patches temporarily in the ports tree

[wsfulton]

There are no plans for another 2.0 release. The next release of SWIG is 3.0.0 and the fix is in master ready for this release. The primary focus for 3.0.0 is c++11 and nested structs so you should probably be upgrading to 3.0.0 when it comes out if you want c++11 support. Is there a problem upgrading to 3.0.0?

[FlorianFranzen]

Well, apart from the fact that there is no stable version available, there is none. Master seems to run just fine on my mac (apart from building the documentation, which is due to a bug in yodl on OS X).

I just thought it would be nice if this feature was backported. Assuming that you use semantic versioning, a step from 2.0.0 to 3.0.0 sounds like a lot of stuff might break and that there are some backward compatibility issues. Since I consider c++11 support an important feature, I seems to be weird to force user to upgrade to 3.0.0 and make them rewrite their code instead of just merging three little patches.

[wsfulton]

Agreed that it would be nice to backport it. If there were more developers we could probably find the time for more releases, but help is in short supply. There is quite a reasonably big effort required in making a new release. I could do one, if there was some sponsorship available or more help about in getting version 3 ready. I do this all in my spare time. Very little should break in version 3. It has been developed with the same high degree of backwards compatibility in mind as other minor releases, with some possible tweaks for nested classes being needed and support for .NET 1.1 was dropped. You could help by trying it out and reporting/fix any issues. It should be released within a few weeks.

[koobs]

I can help by creating a devel/swig30 FreeBSD port utilising the github repository sources in lieu of a future release. If someone could help facilitate by creating the appropriate tags for me to fetch against, that would be fantastic. A unmoving named target is preferable to a commit hash.

This will at least get the current sources into the hands of FreeBSD developers, port maintainers and users in a convenient-to-play-with form, and at most will identify issues that can be sorted out prior to release.

A tag like an 3.0.0-rc1 (or beta1 if its not yet RC-ready) should be representative enough to work with

[wsfulton]

@koobs, we need to make one more change in the nested class area then it will be ready for a beta release. It will probably be done this weekend as mentioned on the swig-devel mailing list. I will create a label for you to use when done.

[wsfulton]

There was quite a lot of interest in a 2.0.12 release, so I've spent the weekend creating, testing and releasing it. It should contain the fixes you need.

I'll do a tag in a day or two for 3.0.0

[koobs]

Much appreciated William :)

[wsfulton]

I've created a rel-3.0.0-beta1 tag. There aren't going to be any major changes until the release apart from some Lua changes. Any feedback from FreeBSD testing will be appreciated.