Update predefined versions

[jpf91]

There wasn't much feedback on the newsgroup so I've implemented the suggested changes. Please give some feedback if those changes are OK.

• Consistent naming: SoftFP --> SoftFloat (on ARM SoftFP != SoftFloat)
  HardFP --> HardFloat
• DOC: ARM_Thumb means any Thumb version
• ARM/ARM64 fix reference regarding AARCH (AARCH is an arm8 term
  but ARM is also defined for ARM7,6,5...)
• Remove MIPS: Use MIPS32 instead
• Remove MIPS_NoFloat: This should be a generic version D_NoFloat
  or it shouldn't be defined at all.
• D_X32: Add a note that defining both D_X32 and X86_64 is
  valid/expected
• D_LP64: We only talk about pointers, this is not the C LP64 data
  model!

[alexrp]

cc @dawgfoto

[ghost]

D_LP64: Technically there is no long long or "C" sizes in D. However it does affect the aliases in druntime such as core.stdc.stdint and core.stdc.config.

You could just say it only affects the pointer size and nothing else.

[jpf91]

core.stdc.config doesn't use D_LP64 and core.stdc.stdint uses it only as "is pointer size > 64".

The problem is that LP64 has actually a different meaning in the C world: There it does also mean that "long" is 64bit so some people could expect that D_LP64 means that the matching C compiler on that architecture would use 64bit longs.
http://en.wikipedia.org/wiki/64-bit_computing#64-bit_data_models
GDC currently even uses that definition and therefore doesn't define D_LP64 on ILP64 systems. So I think we should make it very clear that D_LP64 only gives information about pointer sizes and nothing else.

But probably I should really formulate it in a shorter way.

Regarding the definitions in core.stdc.config: I think if we try to support all of LLP64 LP64 ILP64 and SILP64 the current approach will be annoying. We actually need c_int and even c_short as well for proper C interop.
I like the gdc solution a lot: Expose __builtin_clong, __builtin_culong etc types and make core.stdc.config use aliases:
https://github.com/D-Programming-GDC/GDC/blob/master/libphobos/libdruntime/core/stdc/config.d#L24

The only place where we need to know about LLP64 LP64 ILP64 and SILP64 in D are those definitions in core.stdc.config. So instead of adding new versions for those using the GDC way with builtin types seems to be better.

[MartinNowak]

Remove MIPS32: We should have *32 variants either for all architectures or for none.

I'd rather like to see a transition to X86_32.

[jpf91]

Strange X86 isn't defined on X86_64 by dmd. I always thought dmd sets both - has that changed or have I made that up?
Anyway, doesn't that mean that we don't need the _32 versions at all as the normal versions mean the same?
And why is MIPS the only version which hasn't "32bit" in it's description?

Also updated the D_LP64 description, I hope it's better now.

[MartinNowak]

You made it up.
X86 means X86_32.
A third of druntime's version statements do this.

version (X86) version = AnyX86;
version (X86_64) version = AnyX86;

I don't think we should repeat this for every architecture we're adding.
There was a discussion about this in #207.

[jpf91]

Um I agree but I don't know what would be a clean solution. Having the versions without _32 mean 32bit for everything and making a single exception for MIPS is quite confusing. And I don't think we can change it for all versions though as this would break x86 code.

I'd personally continue defining AnyMIPS manually and lobby for version expressions. Sooner or later we'll need those anyway.

Or maybe we can sneak in a __traits(version) into dmd so we could wrap it in a template and do things like this:

static if(isVersion!"MIPS" || isVersion!"MIPS64")
{
}

[MartinNowak]

single exception for MIPS

It's ought to be for all architectures.
I hope that X86 was the only exception before we learned better.

Even if we have recombinations the fundamental issue remains. A version block performs an intersection so we'll often need the complete set to start with. If we don't have that, one has to list all architectures, always. Lots of useless repetition for the most common case.
After doing the MIPS port I have a really strong opinion on this, because the current approach clearly doesn't scale well. We're defining version identifiers, they should fit the need.
It also maps much better to C.

this would break x86 code

We have a proved solution to this.
Now X86_32 as synonym, 6 month later a X86 warning and another 6 month later a X86 deprecation.

[MartinNowak]

The only argument I can come up with to favor combination is the one Walter uses to chose
#if linux || __APPLE__ || __FreeBSD__ || __OpenBSD__ || __sun when
it's actually #if POSIX. The compiler will show you where to check code for a new port.

[WalterBright]

Using X86 is traditional for 32 bit code on the x86 processors. Using X86_64 is traditional for, you guessed it, 64 bit code on the x86 processors. I don't see any compelling reason to change these.

[jpf91]

A compelling reason would be too keep version identifiers consistent, so If MIPS means MIPS32/MIPS64 then X86 should also mean X86_32 / X86_64 or we'll soon have a similar mess with version identifiers as in C. And @dawgfoto is right that in most cases you don't care whether the architecture is 64 bit or not so you'll have to add boilerplate Any_arch code a lot.

There's not really a traditional meaning, every C compiler has a different predefined macro for X86*. But in C you can at least do #if defined __x86_64__ || defined _X86_. The same code in D adds much more boilerplate so those traditional meanings don't fit the D version system well.

** http://sourceforge.net/p/predef/wiki/Architectures/

Deprecating X86 could be a solution, but at some point X86 would come back with a different meaning - this process sounds really painful and I can already hear the complaints.

Maybe we need "complete sets" in general, but we could make an exception for architectures as there are only two choices. You could always get the complete set with version(X86 || X86_64) as in C. This is imho the nicest solution as we don't have to deprecate anything. But it requires allowing expressions in version statements.

Or we add Any_ARCH versions to the compiler. The naming is not really good but at least we don't need to deprecate any version identifiers.

[alexrp]

I think enough use cases for Boolean expressions in version statements have been presented thoughout recent months (years?). I think (very strongly) that we should support them. Not only are there use cases for this, there is actual evidence that the current system is not good enough.

[MartinNowak]

Walter you are right about the historical meaning of X86, maybe I am off the track here and focused too much on consistency. I think we will benefit from staying close to C tradition with this.
But to rephrase my main point.
What I want is a single version identifier per architecture family.
This is useful in may places and one level above architectures.
We are already using the AnyX idiom by convention, it is very descriptive so AnyX86, AnyMIPS, AnyARM and something like D_InlineAsm_AnyX86 might be good candidates.

Indeed we would not lead this discussion if version identifiers were recombinable and could be imported. We would simply add a D module that does the mapping, DRY. If we do not solve this enums will supersede version identifiers at some point. I just noticed I already used this myself to get importability.
I encourage you, Walter, to reread proposal Bugzilla 7417. It is not complete though because it does not tackle the import problem.

In the meantime AnyX would be helpful.

[WalterBright]

I think enough use cases for Boolean expressions in version statements have been presented thoughout recent months (years?). I think (very strongly) that we should support them. Not only are there use cases for this, there is actual evidence that the current system is not good enough.

This is true IF you are trying to use version blocks in the same way one does in C. However, that is the stylistically wrong way to do it in D. The recent attempt to replace in the DMD source code a bunch of the operating system predefines with POSIX was a failure, and the reason for the failure was a fine example of why the C approach is inferior and why it isn't supported in D. (The Boehm GC source code is a extreme, horrific example of how the C approach goes terribly wrong.)

The evil happens when you add an architecture to the A=B||C||D and then blithely assume that all the version(A)'s will now be correct.

They never are.

A far more robust and organized approach is to figure out just what you're trying to abstract away in these version blocks, and then abstract it away as a struct, class, function, or import. Then,

version (B)
feature() { return foo(); }
else version (C)
feature() { return foo(); }
else version (D)
feature() { return creature(); }
else
static assert(0, "version not supported");

This is clear, simple, organized, and forces the programmer adding E to check each one of those, instead of assuming they will work.

Note: feature() can also be a struct, a class, an import, etc.

If your code is a mess of version declarations, strongly consider rethinking and refactoring.

(One of my goals is to remove all the #if's from the dmd source code using this technique.)

[alexrp]

You are talking about platforms. We are talking about architectures.

The latter are much less likely to break when generalizations are made because all relevant ISA designs from the last 20 or so years are extremely focused on backwards compatibility. It is a simple fact that a lot of inline assembly written for a 32-bit architecture will run perfectly fine on the 64-bit equivalent, for instance.

It is also worth noting that version identifiers can be set on the command line by users of the language. We cannot make questionable decisions about how people must use them just because of short-sightedness.

[WalterBright]

DRY was mentioned. In this context, I believe it is a false god. Operating systems do not change in lock step with each other, despite purporting to present the same interface, They never do, exactly, and things have to be exact. Making adjustments for one system must not break other systems that use the same version, and the person doing those changes rarely (or frankly, never) checks them all, causing bit rot.

Time to throw the C style of versioning under the bus.

[WalterBright]

You're right that code is likely to run, but it definitely does not behave the same in terms of performance. Different CPUs in the same line suggest different instruction mixes, sometimes very different mixes. And yes, there are sometimes subtle differences in the execution results.

Trying to save a few bytes, or even many K bytes, of source code is not a worthy goal.

I know that D users will continue to try and write C style version blocks, but I will continue to try and show a better way. I also am painfully aware that druntime/phobos have turned to C style versioning, despite my exhortations. It should be fixed, not embraced.

[MartinNowak]

Please let's organize the discussion.

The version expression stuff should go to Bugzilla 7417, maybe we should archive the side discussion.

[MartinNowak]

This pull request affects mapping these architectures+x86+x86_64 onto this kind of problem.
The reason I want those architecture families is that glibc/linux has arch-trees where we use version (arch) branches. It's the same with FreeBSD, they copy platform dependent C headers to "/usr/include/machine" at installation time we keep everything in one place.

[jpf91]

So about this pull request: Shall we wait for results of the bugzilla discussion or shall I update this pull request with Any[ARCH] versions? @dawgfoto do you know why glibc uses mips for 64 bit and 32bit, but arm has arm and aarch64?

[MartinNowak]

It's all about MIPS for kernel constants, they managed to use those consistently across all asm code.
The other topic is which one of the crazy O32/N32/N64/O64 ABIs you're using. With aarch64 I don't really know, maybe they are having ABI reasons or the existing code is hard to port to 64-bit.
Walter is right about this being a social/historial issue, e.g. linux did their i386/x86_64 merge.
I was already pretty uneasy with #178 because we committed to tons of version identifiers without practical evidence.
Maybe we should step back a little and remove those again. AFAIK GDC is leading this effort but maybe @klickverbot also has some opinions about that.
How about adding preliminary identifiers prefixed with an '_' so that you can finish you're GDC patch and encourage further porting efforts.
Whenever we do a full port we could then immediately deprecate those.
I think it pretty safe to assume that we need names for the arch branches of common OS/libcs.
But speculating about the details of ARM_SoftFP, ARM_HardFP, D_SoftFloat... is a waste of time until someone really needs them IMO.

[dnadlinger]

@dawgfoto: I don't have a strong opinion on the topic and will merge any patches to make LDC conform to the official list. That being said, I do think that we should come up with a consistent, well-designed set of identifiers for the various systems. And do it soon, because there is actually work going on concerning platform support – @jpf91 is (was?) working on ARM support for GDC, @redstar is porting LDC to Linux/PPC64, …

In short, I think we should put effort into working out a good design now, because changes are still relatively painless, but they might no longer be soon.

By the way, the »correct« way of introducing GDC-specific versions would be using a vendor prefix (GDC_) , not _.

[dnadlinger]

I'm not really familiar with ARM, so please pardon my ignorance, but: Is this actually needed? What kind of code would be specific to Thumb, but work on both versions?

[jpf91]

Thumb1 is a strict subset of Thumb2 so any Thumb1 code would work in both variants. I do not really care about Thumb mode right now though and even if we supported Thumb mode, Thumb1 should be good enough to implement almost everything. The problem is how do we mark ARM non-Thumb assembly correctly (and it shouldn't break even if a Thumb3 is introduced at some point). But I guess if we make it clear that ARM_Thumb is set for every Thumb variant we won't need ARM_Thumb1 ARM_Thumb2 for now.

version(ARM_Thumb)
else version(ARM)

EDIT: (I didn't know that though when I initially wrote this pull request so I'll update it and remove Thumb1/Thumb2)

[jpf91]

And @dawgfoto is working on MIPS ;-) Regarding ARM work I'll continue that soon. I wanted to get the unit tests running on gdc-4.7 first though. I think LDC is in better shape in that regard, gdc still doesn't pass all unit tests or the test suite.

Regarding vendor specific versions:
GDC could map all predefined C macros to boolean values which would then be used with static if. That should be good enough to experiment with different versions but I think we also need a good solution for the standard version identifiers described on dlang.org. My main concern is that those should be consistent and this is mostly what this pull request tries to address.

@dawgfoto I think the ARM_SoftFP etc versions were requested for std.simd, but I might be wrong. I don't think those floating point definitions do any harm though. The names are clear and unlikely to conflict with any other name. In the worst case some of those are redundant.

The real problem is the X86/X86_32/X86_64/AnyX86 problem. We can't be compatible to C and consistent (as C is not consistent: __MIPS__ for mips32/64, but _X86_ for i386 only), so I think being consistent is more important.
Maybe having architecture families be contained in one version as it is for MIPS now is really the best solution, but would changes to the meaning of X86 get accepted?
The other solution is AnyMIPS, FamilyMIPS/MIPSFamily, ArchMIPS or something like that.

I don't really have a strong opinion on how to implement this but I'd really like all version identifiers to have a consistent and clear meaning.

[MartinNowak]

but would changes to the meaning of X86 get accepted

Probably not because as Walter correctly stated it has a traditional meaning outside of D.

The other solution is AnyMIPS, FamilyMIPS/MIPSFamily, ArchMIPS or something like that.

I think we should go with AnyMIPS/AnyX86/AnyARM then as it already emerged in existing Code.
So probably something along this line would make sense then.

AnyMIPS = MIPS32 || MIPS64
AnyARM = ARM || AARCH64
AnyX86 = X86 || X86_64

[jpf91]

I've added the Any* versions, please review again.

[MartinNowak]

I would prefer to use MIPS32 (and probably AArch64?) because that's their "official" name. There isn't much consistency in S390 and S390X either.
As was already mentioned we're still at a point where change is fairly painless so anything from here on is fine with me.

MIPS32
AArch64

[alexrp]

Should be AArch64 though, not all upper case.

[MartinNowak]

thx

[andralex]

So this is okay to merge regardless of the side discussions I reckon. Is that right?

[alexrp]

No, @jpf91 still needs to make a few changes

[jpf91]

Changed MIPS --> MIPS32, ARM64 --> AArch64.

[WalterBright]

I do not understand what the AnyX86 is for. I don't see any use at all for it.

[jpf91]

OK, let's see a real world example:
https://github.com/dawgfoto/druntime/blob/4bde1864bc81cb06e185d100e9c8cc8e9a7315dd/src/core/sys/posix/fcntl.d#L104

version(MIPS) in there is the same as version(AnyMIPS) with this proposal . This proposal only suggests renaming it to have consistent names for all architectures as we can't change the meaning of X86 etc.

These declarations

enum O_CREAT = 0x0100;
enum O_EXCL = 0x0400;
enum O_NOCTTY = 0x0800;
enum O_TRUNC = 0x0200;

enum O_APPEND = 0x0008;
enum O_DSYNC = O_SYNC;
enum O_NONBLOCK = 0x0080;
enum O_RSYNC = O_SYNC;
enum O_SYNC = 0x0010;

are identical on MIPS32 and MIPS64. They are guaranteed to be identical in any case as the C library does not make a difference between MIPS32 and MIPS64 in this case.

Now I challenge you to write the above declarations so that they are:

• only used for MIPS32 and MIPS64 but not other architectures
• without version(MIPS32 || MIPS64) which would be a nice solution if it worked
• without AnyMIPS
• without declaring AnyMIPS yourself (this is not a real solution: You'd have to duplicate the declaration in every module where you wanted to use it. Also the syntax for doing this is ugly)
• without duplicating the code. We usually agree that duplicating code is a bad thing. In some cases there may be much more code than these 10 lines of code. Now if you create a duplicated code block then fix a bug in one block and forget to fix it in the other block this is soon going to be maintenance hell. There are similar situations with much more complicated code where it'd be easy to miss a fix in one code block. So in short code duplication is not an option.

This is not an artificial problem. Anyone porting druntime/D to a new architecture will face it and it is very annoying. You keep telling us that we're using version in the wrong (C) way, but then please tell us how it's supposed to be done correctly.

Note that the solution you stated above ("abstract it away") is not really possible in this case.

I also have to doubt this statement:

The evil happens when you add an architecture to the A=B||C||D and then blithely assume that all the version(A)'s will now be correct.

This may be true in some cases and of course everyone should carefully check all affected code before adding an architecture in such a definition. But in that case the C approach if(B||C||D) is actually safer than what's usually done in D (defining a custom version) as the C approach is always local to the if statement and it's obvious what code is affected but the D custom versions can be spread all over the file.

Your solution is exchanging the problem of wrongly added versions by programmer sloppiness with code duplication and a maintenance mess. I doubt the maintenance mess is better. In druntime/phobos/dmd code we can enforce this but 3rd party libraries will just use static if instead of version statements whenever possible if we don't make version statements a little more pleasant to work with.

(And this is only about combining versions, the fact that versions can not be imported deserves an extra post. Even gdc cheats because of that and the only reason version statements work for architectures is because they're defined by the compiler. Manually defining a custom version is basically useless/impossible right now and even outright dangerous if the version statement affects the ABI)

[WalterBright]

Here's one way to do it:

module foo;

version (linux)
{
public import linux.foo;
}
else version (Win32)
{
public import win32.foo;
}
else version (FreeBSD)
{
public import freebsd.foo;
}
else version (OSX)
{
public import osx.foo;
}
else
{
static assert(0);
}

[don-clugston-sociomantic]

@walter - That's ugly, and it hides the fact that most of these values are exactly the same on all systems. With my proposal from 7417 it would be:
version AnyMIPS = MIPS32 | MIPS64;

version (AnyMIPS) { ... }
Fixes all this stuff, avoiding the incomprehensible mess you get with C, and the birds nest of version statements you get in D at the moment.

[WalterBright]

The salient point is "most". Not all. I've found time and again that different systems have the same names, but one or two of the values are different. It's maddening, and it causes subtle and terrible problems when someone decides that SystemA declarations can be merged with SystemB's without going through them line-by-line.

Worse, when SystemA and SystemB share the declarations code, and Fred issues a pull request to fix things for SystemA, then SystemB unwittingly gets broken.

Instead of AnyMIPS, all you would need is:

version (MIPS32)
  public import mips.foo;
else version (MIPS64)
  public import mips.foo;
else version (OSX)
  ...

No, I don't think that is ugly (though of course aesthetic taste varies), all the versioning is done in one import and is not embedded in the various modules themselves, and the expert on OSX can maintain osx.foo without worrying about its impact on any other system.

I've been drifting towards this style for years (you can see it in D1 Phobos in an early form) and I think it works well. I'm sorry that the D2 library has gone back to the C way.

[don-clugston-sociomantic]

Walter - you're basically advocating copy-and-paste. I've done what you suggest several times, and without fail, it has led to a maintenance nightmare.
Please, don't describe what I'm suggesting as "the C way". I think we would indeed get the C way if boolean conditions were allowed in version statements. That encourages a rat's nest of conditions where it's becomes extremely difficult to work out which code is being executed. I'm only advocating them in version declarations. This would force clarity of thinking in versioning, even more than we currently have.

[jpf91]

@WalterBright your solution is flawed: It does work in this case as there are only declarations, but as soon as there is code which must be compiled and not only imported (e.g. a small function replacing a C macro) you have to compile mips.foo.d.

Now how do you make sure mips.foo.d is being compiled on MIP32 AND MIPS64 but on no other architecture? You either have to do build system tricks or you have to add the code duplication in mips.foo.d.

BTW: Your solution is the "glibc" way. Glibc also outsources some declarations into arch directories. But maintaining that is a mess. A similar way would be to use git and have one generic branch, then keep architectures as branches in git. This at least allows easily merging common changes without copy & paste.

[jpf91]

OK, so what should I do to get this pull request accepted? If I removed the Any* commit the remaining changes would only make sure that version identifier names are consistent.

@dawgfoto is that OK with you? This would mean you'd have to declare AnyMIPS by yourself or somehow change your code. I don't see an other option to get consistent names though as we can't seem to reach consensus regarding Any* and similar solutions.

[WalterBright]

@jpf91 I won't argue that my solution is perfect. I'm arguing it's better, and that it's more tasteful. To solve your issue with mips.foo.d, I'd add the following code into it:

version (MIPS32) { } else version (MIPS64) { } else static assert(0, "this is only for MIPS");

No, it ain't beautiful, but it gets the job done with minimal hassle and without throwing out all the other benefits of the approach I suggested.

Don, it's hard for me to respond to your case where things led to a maintenance nightmare without more specifics. It's possible that I have badly explained the idea and you've implemented something else. I can tell you that I've had maintenance nightmares from the old way. Just recently, there was a pull request where all the system macros in dmd were pulled into a POSIX macro, and that was used instead. Well, it bombed in the test suite. Nobody went to figure out why, and it was reverted. That was an easy one, but it was hardly the only problem I've encountered with attempts to save keystrokes by combining declarations from different systems into a common source base. I've repeatedly run into issues where different systems had one of the values for a list of constants be slightly different. Just enough to superficially look the same, so it wound up in the common section, and yet it would fail in mysterious ways. I freakin' hated debugging those.

[MartinNowak]

Any*

If we can't reach consensus then leave them out.

Most of the other things look good.

make sure that version identifier names are consistent

Don't overstress this please.
What happened before is that the X86/X86_64 was applied to MIPS/MIPS64 and ARM/ARM64 which have their own established naming schemes.
Now MIPS32/MIPS64 is fine and ARM/AARCH64 probably is too. But I have no idea about ALPHA, PPC, HPPA, S390, SH etc. and at least PPC64 looks fishy.

[jpf91]

I don't care that much about upper/lower case, what I want to avoid is having two version identifiers which sound like the same but have a different meaning.

I looked up the other architectures and the casing doesn't match most of the time: PPC64 is ppc64 , IA64 is usually IA-64, SPARC64 is correct, S390X (usually lower-case), HPPA64 (usually lower case), SH64 (sometimes lower case).

I would just leave those as is, but I don't mind changing them.

[alexrp]

Is there any reason we don't have MIPS_MEABI as per GCC's mabi=meabi option?

[jpf91]

I only can find mentions of MEABI in gcc-3.3 docs. In more recent releases (http://gcc.gnu.org/onlinedocs/gcc-3.4.1/gcc/MIPS-Options.html http://gcc.gnu.org/onlinedocs/gcc-4.7.2/gcc/MIPS-Options.html) MEABI isn't mentioned anymore. I have no clue what happened to it though.

[alexrp]

Related question: Since the EABI (the one that hasn't mysteriously disappeared) has 32-bit and 64-bit variants, should we have MIPS_EABI32 and MIPS_EABI64?

[jpf91]

I guess the intent with the MIPS ABIs was to follow gcc which defines _ABIO32 and _ABIO64 but only __mips_eabi for both EABIS. I'm not sure why gcc does it that way though. @dawgfoto could MIPS_O64 and MIPS_O32 be replaced by ````MIPS_OABI+D_LP64` or is that a bad idea? Should we introduce `EABI32` and `EABI64`?

[MartinNowak]

Anything that does not map 1-to-1 to the ABIs and C defines is a mistake in my opinion. I think O64 is ILP32 btw.
OTOH using D_LP64 with EABI would be OK.
http://www.linux-mips.org/wiki/MIPS_ABI_History

[jpf91]

OK, can we merge this then? Any further comments?

[MartinNowak]

Please merge