How about integrating with Alien::TinyCC ?

[aero]

Hi

tcc source is bundled in XS::TCC.

There are another tcc-perl binding modules ( Alien::TinyCC, C::TinyCompiler )
Now Alien::TinyCC can also be built on Windows. ( run4flat/Alien-TinyCC#1 )

How about integrating & building with Alien::TinyCC like C::TinyCompiler instead of bundling tcc ?

[run4flat]

@tsee, you and I are thinking along the same lines. :-)

I've gotten some basics started, including a potentially over-engineered package system to build reusable modules. For example, I can call C-compiled functions from Perl, and I've added a simple data structure system. But I know I could do a lot better, especially with calling compiled functions from Perl.

Shall we try to merge XS::TCC and C::TinyCompiler?

David

[tsee]

Hi David,

On 08/21/2013 05:58 PM, David Mertens wrote:

@tsee https://github.com/tsee, you and I are thinking along the same
lines. :-)

I've gotten some basics started, including a potentially over-engineered
package system to build reusable modules. For example, I can call
C-compiled functions from Perl, and I've added a simple data structure
system. But I know I could do a lot better, especially with calling
compiled functions from Perl.

Shall we try to merge XS::TCC and C::TinyCompiler?

tl;dr: I'm open to that, with some caveats.

First a quick disclosure: XS::TCC has been on github for a while. It was
only when Rafael sent me an email saying "look here, didn't you do this
a while ago?" that I saw your blog post yesterday afternoon. Originally,
I hadn't pushed XS::TCC to CPAN because there's a few shaky details and
a few significant problems that made me think I wouldn't yet risk my
$work's infrastructure on it. But $work's requirements are way beyond
those of most other uses of Perl and employers relying on Perl, so I
took the hint to get over myself. So no. Not coincidence that they got
released around the same time. I wish I'd done it sooner since that
might have avoided a good amount of duplicated work.

Fundamentally, the two modules work a tad differently. You rely on "FFI"
and packing complex data structures. Have you seen the
Convert::Binary::C module? It goes to great lengths to make such
endeavours even possible for complex data structures that may suffer
from alignment and other fun things. XS::TCC strikes different
trade-offs (but trade-offs still). It uses ExtUtils::Typemaps, a module
that was originally refactored out of ExtUtils::ParseXS/xsubpp, to
statically generate and compile the code to translate input parameters
and return values between Perl and C. This means that it's a lot easier
to transform complex types, but at the cost of understanding yet another
arcane mechanism.

On top of that, there is a very curious effect that is specific to the
Perl API and TCC. TCC doesn't optimize very well. The Perl API consists
primarily of macros. The combination is very unfortunate. Where
normally, a C compiler that can reach optimization levels of gcc's -O or
-O2 modes can eliminate a large fraction of the code that the macros
expand to. TCC can not. That means that suddenly what started as an
optimization (macros == inlining) becomes a drag. If, instead, the Perl
API was primarily functions (compiled by the system compiler), then
XS::TCC (and also your module depending on how far you go) would be a
lot more efficient at marshalling data.

To wit, some arbitrary n^2 thing I used for testing showed TCC to beat
Perl by about 10x. For, say, matrix multiplication where you have a
large amount of marshalling compared to the actual work, this looks more
like a head-to-head race between perl executing as
C-code-compiled-by-optimizing-compiler and TCC executing as
C-code-compiled-by-TCC.

The way out for many typemaps is to include functions in the actual
TCC-wrapping module that are compiled by the system compiler. I did that
for a few basics. But how do you take advantage of that for your own
custom typemaps? That's really only possible if one were to wrap a good
chunk of the Perl API in C functions and ship those with the module for
use in typemaps. For things that are already opaque C pointers to Perl,
such as PDLs, this shouldn't be nearly as much of a problem.

You've had a some really good ideas that I'd love to see in a merged
module. For example, IIRC, XS::TCC will attempt to expose all C
functions it finds to Perl unless there's explicit options to tcc_inline
that list the functions to expose. Annotations in the C code are much
better.

Did you solve the debugging-this-inlined-C-code-is-a-nightmare-problem,
by the way? For me, at least line numbers were still off and some
failures were just completely opaque. I think those were commonly very
bizarre linking issues. Also, reusing the same TCC compiler object was
occasionally "interesting" since not all headers involved seem to have
conventional #ifdef guards against multiple-inclusion.

This was about where I left off and started being distracted by my PLua
project to embed luajit in Perl with very tight integration (to the
point of being able to access Perl lexicals directly from embedded Lua
blocks)[1].

This in turn has most recently been pushed down in priority when Mattia
Barbon convinced me that if I want speed, making Perl faster is much
better than a cop out such as deferring slow bits to another language.
So we revived my Perl::JIT experiments[2] and have been hacking away on
that since. (The joy of working with some of the best devs to be found
anywhere at your day job :))

The last bit is important only because I currently can't help much with
merging the two approaches and polishing the result. The JIT is as
ambitious as it is exhilarating and I plan to focus my little spare time
on that for the foreseeable future. Nonetheless, I'm open to merging the
two modules. I have some reservations about requiring packing data
structures for the transition, though. Please do read the
Convert::Binary::C docs. :)

Either way I've given you commit/push access to the XS::TCC repository
and have no qualms about adding you as PAUSE co-maintainer as well.

Best regards,
Steffen

[1] https://github.com/tsee/p5-Plua
[2] https://github.com/tsee/jit-experiments,
https://github.com/mbarbon/perl-libjit, https://github.com/mbarbon/b-replace

[tsee]

BTW, calling this kind of setup a "JIT" is a bit misleading, IMO. :)

[tsee]

Also, I'm by all means open to using Alien::TCC. Just not in a position to do the work. See also @tokuhirom's tickets about the TCC compilation/compatibility with OSX.

[azawawi]

@tsee++ @run4flat++ 👍

[run4flat]

Lots to reply to. I know what you mean about things sitting around and not getting pushed out to CPAN. I've been working on C::TinyCompiler in spurts for about two years. I polished it and pushed it out only recently in reply to an email on the tincc-devel mailing list.

First the easy one: line numbering. I wrote a function called "line_number" that generates a "#line" directive for the user: http://search.cpan.org/~dcmertens/C-TinyCompiler-0.02/lib/C/TinyCompiler.pm#line_number. (I apologize for giving a link to a specific version of the module; I'd use metacpan, but it's displaying my README.pod instead of my module's pod, for some irritating reason.) This was based on some work I did with PDL::PP, and the lack of line reporting in that module's output. I could do better by adding another API function for appending code to the Body section which gets the line number from caller().

As for the meatier question: I think we really are mostly aiming at the same mark, but we have an important difference in our approach. The key difference is that I explicitly did not want to require the Perl API headers in all compilation contexts. I want to give the minimal amount of code to tcc when it comes time to compile, and I want to compile as much as possible at build time, with an optimized compiler. Two motivating examples: (1) using the GSL and (2) wishing to construct a C function for a C callback (i.e. not a Perl function reference callback wrapper, a real C function). For GSL, I don't want tcc to spend time reading the C sources from disk and building unoptimized machine code when I already have an optimized .so or .dll. And if somebody is trying to write a simple C function callback, they don't need the Perl headers, so why force them? These examples, and the goal for minimal C code in the context, led me to develop the package system, which (among other things) aims to make it easy to add pre-compiled functions to a compilation unit.

In fact, I really don't care much about data munging because, as you said, I would rather folks use Convert::Binary::C for those purposes. I suspect you got that impression after reading over the poorly internally documented C::TinyCompiler::Callable.

That brings us to FFI. Although at one stage in its development I decided to fall back on FFI::Raw for invoking the C functions, I decided that FFI::Raw's interface was too closed for my needs to be useful. For example, I had no simple way to send PDL data to a function with an FFI::Raw interface, even though I could get a reference to the SV* that held the PDL data. I also didn't like that FFI::Raw depended on an external library, but didn't provide for it. For these reasons, I wrote my own function wrapper code. It's slow and kludgey, and it really should use typemaps, but it works for now with basic types and pointers. The flexibility with the pointers is really the big win. Still, my hope is to eventually replace it with a bona fide xsub wrapper, and to utilize the institutional knowledge already wrapped in known typemaps. Fortunately, my little ffi is just a "C::TinyCompiler::package," with a simple enough API, and it should be possible to completely alter the internals to use xsubs once I figure out how to distill Perl's xsub signature, and use the typemap code.

Originally I had hoped to provide a direct interface to libtcc. However, I quickly discovered with a few tests that two uncompiled compiler contexts cannot coexist in tcc: it uses internal globals during the compilation stage. For this reason, I changed my approach. I don't actually create a TCCState until the Perl-side "compile" method is invoked. All of my Perl-side compiler object methods simply add data to the blessed hash. When it comes time to compile, then I do any source munging, create the TCCState, compile, and relocate. This way, you can have multiple compilers floating around without clashing. The only potential pitfall now would be due to simultaneous compilation in a multithreaded environment, or a package that stupidly decides to compile something during its preprocess stage.

As to the topic of this ticket, I'll work on a branch to use Alien::TinyCC and submit that as a pull request in a bit. I'll keep it as a pull request, though, so that you and anybody else interested can independently clone the work and test it before it gets merged.

[azawawi]

Indeed a big reply 👍

I am watching this project more closely.

[run4flat]

By the way, it may be possible to completely avoid any Perl-side function parsing. See http://lists.nongnu.org/archive/html/tinycc-devel/2013-08/msg00019.html, and the follow-up with the actual patch at http://lists.nongnu.org/archive/html/tinycc-devel/2013-08/msg00020.html

[tsee]

XS::TCC 0.03 now relies on Alien::TinyCC!

Cheers and thanks for the help and encouragement!

Closing this ticket since it pertains to using the Alien module specifically.

[run4flat]

I just realized the -fPIC stuff is only in git, so I'll push out a non-dev
v0.04 release shortly.

David

On Thu, Aug 29, 2013 at 1:46 PM, Steffen Müller notifications@github.comwrote:

XS::TCC 0.03 now relies on Alien::TinyCC!

Cheers and thanks for the help and encouragement!

Closing this ticket since it pertains to using the Alien module
specifically.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1#issuecomment-23508468
.

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." -- Brian Kernighan

[azawawi]

@tsee++ @run4flat++ 💯/💯

How about using Travis CI?