[new AA] implement core.internal.aa

[IgorStepanov]

This PR introduces new AA implementation. At the first stage, compiler will not know about core.internal.aa.AssociativeArray and will use only object.aaNew and object.aaLiteral methods.
Thus, this AA implenetation should replace rt.aaA implemetation.
When first stage will be finished, we will start work to endowing core.internal.aa.AssociativeArray with all the abilities of language's AA. For example:

int[string][string] table;
table["foo"]["bar"] = 3;

AssociativeArray!(string, AssociativeArray!(string, int)) table2;
table2["foo"]["bar"] = 3; //exeption, because "foo" doesn't exist in table2

However, now I dont's talk about second stage. I want to finish first stage, wait one release cycle (to catching all possible troubles), and start to work under second stage when the next D version (after first stage finish) will be released.

[IgorStepanov]

@MartinNowak this PR need a you review. New AA has tested with this AA implementation. Passed All druntime, phobos tests and dmd tests (except tests, which disabled in D-Programming-Language/dmd/3904).

[IgorStepanov]

@MartinNowak ping

[MartinNowak]

OK, will review it in the next days, but I'm quite busy right now :(.
I would still stick with my suggestion to use AA!(Key, Value) aaLiteral(Key, Value)(AATuple!(Key, Value)[] data...) for the compiler->runtime interface. It avoids the template bloat of T... and also leave the type deduction in the compiler.

[IgorStepanov]

OK.

I would still stick with my suggestion to use AA!(Key, Value) aaLiteral(Key, Value)(AATuple!(Key, Value)[] data...) for the compiler->runtime interface. It avoids the template bloat of T... and also leave the type deduction in the compiler.

Type deduction is performed by compiler now. I add a type deduction code for future use.
I fear to do big changes in compiler code at a time. In AA!(Key, Value) aaLiteral(Key, Value)(AATuple!(Key, Value)[] data...) we need to implement implicit casting from AA literal to user AA!(K, V) type.
Also, we can't create user AA, which has language AA behaivour (except literal and .stringof). e. g. language AA index operator has a more complex behaivour then we can implement through opIndex.
P.S. also, please see #932 PR. In has been previously reviewed by @DmitryOlshansky and contains minor changes in internal.hash module.

[MartinNowak]

e. g. language AA index operator has a more complex behaivour then we can implement through opIndex.

Well opIndex can either assign or a construct, but this could be done in a UDT as well.

I worked on the compiler side (see here) and my biggest issue was that the type of an AA literal is also inferred from the LHS of an assignment.

static assert(is(typeof(["a": 0, "b": 1, "c": 2]) == int[string]));
ubyte[string] aa = ["a": 0, "b": 1, "c": 2];

[IgorStepanov]

Yes, i understood. I simly want to implement AA stuff with minimal compiler changes, to reduce atomic changes. This way potentially reduces regression count before the next release.
After that we will able to trust to AssociativeArray implementation and start big changes in compiler, like literal handling, opIndex and other ones.

[IgorStepanov]

What does a stage of your libraryAA now? Does it passes most of the tests? (Yes, multiply index, aa.string of and similar will be failed, but what about other tests)?
There will be troubles with is expression, partial specialization template foo(T: int[K]). Have you solved it?

[MartinNowak]

There will be troubles with is expression, partial specialization template foo(T: int[K]). Have you solved it?

That's easy, int[K] is lowered to AA!(K, int) and template foo(T : AA!(int, K), K) already works as expected. So the point here is is(int[K] == AA!(int, K)).

[MartinNowak]

In AA!(Key, Value) aaLiteral(Key, Value)(AATuple!(Key, Value)[] data...) we need to implement implicit casting from AA literal to user AA!(K, V) type.

We just need a way for the compiler to pass the data and I think AATuple!(Key, Value)[] data... is a very good one for the following reasons.

• Data is passed on the stack
• Least amount of template bloat
• We can later use the same compiler mechanism to allow efficiently passing AA literals to other hashmap implementations

[yebblies]

AATuple!(Key, Value)[] data...

Don't you mean Key[] keys, Value[] values? Typesafe variadics loose their advantages over multiple arrays when the compiler is the only one that calls them, and not having to define new type is nice. I can't remember what we decided at dconf.

[IgorStepanov]

We just need a way for the compiler to pass the data and I think AATuple!(Key, Value)[] data... is a very good one for the following reasons.

This PR is not about aaLiteral, but about AssociativeArray.
I can rework dmd part to use another aaLiteral, while you are reviewing this PR.
Should I do that? What aaLiteral signature we prefer? With AATuple or with two arrays, as @yebblies suggests?
Or my old version:

struct AAPair(Key, Value)
{
   Key key;
   Value value;
}
template isAAPair(T)
auto aaLiteral(T...)(T args) if(each!(T, isAAPair)) //args is a tuple of pairs.
{
}

My version allows aa literal with different types of key and value. In can be used for containers like JSONObject:
JSONObject obj = ["key1": null, "key2": 42.0, "key3":[" foo":"bar"]];

[yebblies]

Should I do that? What aaLiteral signature we prefer? With AATuple or with two arrays, as @yebblies suggests?

It actually doesn't matter for now, so long as it works. So long as it is completely internal to the compiler+runtime, we can change it at will. Implement whatever is easiest.

[IgorStepanov]

It actually doesn't matter for now, so long as it works. So long as it is completely internal to the compiler+runtime, we can change it at will. Implement whatever is easiest.

My current implementation is the easiest, as I think. However, the next our goal is the implementing AA as library type and @MartinNowak suggests to do a next step now.
BTW. Current implementation doesn't lower AssocArrayLiteralExpression to aaLiteral during semantic, because we can't use AssociativeArray in CTFE (small differents with native AA, which I mention early). AssocArrayLiteralExpression::semantic prepares aaLiteral call expression and it is wrote to object code directly in toElem/toDt. If we want to lower AssocArrayLiteralExpression to aaLiteral call expression, we should find a way to implement AssociativeArray interface similar native AA.

[IgorStepanov]

@MartinNowak ping.
I'm avaiting two you actions:

1. Review this PR. (When you can find a time)
2. Approve AA construction scheme with aaLiteral, or suggest another scheme, considering my arguments. (As possible faster)
   Thanks.

[MartinNowak]

It actually doesn't matter for now, so long as it works. So long as it is completely internal to the compiler+runtime, we can change it at will. Implement whatever is easiest.

True that.
AATuple!(Key, Value)[] is better suited for passing AA literals to user functions, because a simple implicit conversion rule would suffice to support that.

[MartinNowak]

We can also reuse AATuple for the requested byPair function.

[MartinNowak]

pass key and value by ref

[IgorStepanov]

Ok

[IgorStepanov]

I remembered why I pass arguments by pointer.
In some places (like line 561) I need to cast argument from const to mutable. When I cast l-value, it becomes r-value: e.key is l-value, cast(Key)&e.key is r-value. Pointers helps avoid this trouble.

[IgorStepanov]

True that.

New literal syntax implementation can has a pitfalls. For example: AATuple!(Key, Value)[] passing may cause redundant postblit calls.
We should do that, but at current stage it isn't necessary.
I suggest the next plan:

1. Move to the new AA implemenation: [new AA] implement aaLiteral #933 [new AA] implement core.internal.aa #934 [new AA] aaLiteral support dmd#3904 and final druntime PR (not commited yet)
2. Implement missing parts of AssociativeArray interface like multiple opIndex (may require additional disscussion and language changes)
3. Implement new literal mechanism, move compiler from _aaXXX function calls to lowering V[K] -> AssociativeArray!(K, V) and finish integration of library AssociativeArray: make correct .stringof .mangleof and other.

In this plan, each item depends on previous and in should be executed in series.
If you accept this plan, please review and merge #933. It's very simple (~50 significant rows)

[MartinNowak]

This prime stuff is unnecessary when using a proper hash. I'd favour a pow-2 hashtab as in rt.util.container.hashtab, which is simpler and cheaper to reshape.

[IgorStepanov]

Ok

[MartinNowak]

But if it turns out to be too much work, leave it aside.

[IgorStepanov]

Done. BTW. Why core.stdc.log2 isn't marked as pure?
This change causes small phobos issue: D-Programming-Language/phobos/pull#2549

[MartinNowak]

We don't deprecate impure toHash. We deprecate calling impure toHash from pure code:

Yes, you're right. That still means we'd possibly break working code, no idea how to solve it though.
We might have to see how big the impact is during the next alpha.

[schveiguy]

We don't deprecate impure toHash. We deprecate calling impure toHash from pure code:

Yes, you're right. That still means we'd possibly break working code, no idea how to solve it though.
We might have to see how big the impact is during the next alpha.

I would point out that Object.toHash is impure, with no really good fix on the horizon. This means you could never use classes as keys in pure code.

The situation is kind of sucky...

[IgorStepanov]

This means you could never use classes as keys in pure code.

This implementation uses template-based hash calculation, thus you may use your class as key (in pure code), if your static type of your class declares pure toHash:

class Foo
{
    override size_t toHash() pure const {...}
}

int test(int[Foo] aa, Foo key) pure
{
     auto p = key in aa;
     ...
}

[schveiguy]

OK, I was under the impression you could not change purity on overriding, but maybe that's only if you go from pure to impure.

[IgorStepanov]

That still means we'd possibly break working code, no idea how to solve it though.
We might have to see how big the impact is during the next alpha.

Ok. However our main goal is a choosing and implementing dmd part. What you have objections against my implementation?

[MartinNowak]

I was working on the dmd part, but the AA implementation doesn't work with non-copyable values.
Try to add the following unittest.

unittest
{
    static struct NC
    {
        @disable this(this);
    }

    AssociativeArray!(int, NC) aa;
    aa[0] = NC();
    aa[1] = NC();
    aa[0] = NC(); // new behavior!!!, should throw unless D_NoBoundsChecks is set
}

Also RValues aren't moved into the AA in the most efficient way, and it's the extra copy that makes the compiler complain about the disabled postblit.

[MartinNowak]

I was working on the dmd part, but the AA implementation doesn't work with non-copyable values.

Please note, non-copyable values are always RValue assignments (the crappy aaGetX API won't let you know). So for copyable RValues we can't do an efficient assignment, but for non-copyable values we have to.

[MartinNowak]

See IgorStepanov#1.
There is still quite some stuff to do to end up with a good implementation.
I wonder if throwing away the old rt.aaA cruft is the better approach here?

[MartinNowak]

So for copyable RValues we can't do an efficient assignment

That's actually quite a bummer. Makes me even wonder whether the whole vtable idea is good.
I mean we/you invest a lot of time for this intermediate solution, that isn't good enough to stick around for very long. While at the same time it remains unclear whether we manage to implement a full library AA!(Key, Value) implementation.

[MartinNowak]

How about this?
We focus on writing an almost perfect AssociativeArray implementation for the next release.
Then people can try it out and adapt their code for the various subtle semantic differences.
Once this is really stable we can migrate (deprecate...) the builtin AA sematics to match those of the library type and after that replace the builtin with the library implementation to get back the syntax sugar.

[IgorStepanov]

We focus on writing an almost perfect AssociativeArray implementation for the next release.
Then people can try it out and adapt their code for the various subtle semantic differences.
Once this is really stable we can migrate (deprecate...) the builtin AA sematics to match those of the library type and after that replace the builtin with the library implementation to get back the syntax sugar.

Hmm, I think, users will not have much motivation to test new AA implementation instead of built-in aa.
Moreover, there are some things which can not be implemented in library type.
We may add a special compiler flag: -oldAA to force use old aa, if new AA will violate someone's code. However I don't see any cases when new AA will work worse/different than the old AA.

[IgorStepanov]

That's actually quite a bummer. Makes me even wonder whether the whole vtable idea is good.
I mean we/you invest a lot of time for this intermediate solution, that isn't good enough to stick around for very long. While at the same time it remains unclear whether we manage to implement a full library AA!(Key, Value) implementation.

You worry too much because of this (non-copybale elements). Current soluthion follows a existing implementation. We may add simple bitwise copy for it (as temporary hack).
After that we may:

1. reject non-copybale elemenst (better solution, I think).
2. implement special infrastructure for non-copybale elements:
   (The main trouble is a passing element without postblit/dtor calling).
   We may use lazy arguments for it: if Value is non-copybale opIndexAssign should be implemented like opIndexAssign(ref Key k, lazy Value val); This signature will accept only literals for non-copybale elems and allow to store it without dtor calling.

How about this?

Still, I would suggest to speed up the process and introduce new AA as possible faster, because it better than existing AA in many respects and I do not see any disadvantages in comparison with the old implementation. Anyway I sure that if this disadvantages will be found, we will able quickly fix them. In other words if we introduce this implementation then nobody gets hurt. Thus there are not reasons to delay this.

[MartinNowak]

Hmm, I think, users will not have much motivation to test new AA implementation instead of built-in aa.

They'll have a very high motivation to do this, the implementation will be faster and literals can be put into the data segment.

Moreover, there are some things which can not be implemented in library type.

Do you have a list of those? I know that ubyte[ushort] = [0: 1, 1: 2]; is problematic, because the compiler uses the left hand side to infer the literal type. In a library one could implement this as AA!(ushort, ubyte, 0, 1, 1, 2)().

We may add a special compiler flag: -oldAA to force use old aa, if new AA will violate someone's code.

We would first have to add a newAA flag.

However I don't see any cases when new AA will work worse/different than the old AA.

This pull uses vtables and generates code for each template instance, that's already worse. On top of that it suddenly disallows things where the old implementation was simply sloppy. This will break code which is really worse.
We need to come up with an AA implementation that is much better than the current to justify all the effort. Breaking code is an absolute no-go, especially since the last 2 releases already broke AA usage (see here).
I'm not sure about this, but it seems to me that a high-quality core.aa module can help us to transition away from the builtin AA.

These would be my requirements for a good generic AA implementation:

1. open addressing
2. efficient construction, insertion and assignment (no extra copies or postblits)
3. fully CTFEable (includes storing literals in the data segment)
4. type and attribute correctness
5. get's rid of TypeInfo methods (toHash, opEquals, tsize)
6. GC NO_SCAN for values

With the vtable approach we can only achieve 1, 5, and 6 and the dmd pull (https://github.com/D-Programming-Language/dmd/pull/4175/files) already adds a lot of code that is specific to the vtable implementation but wouldn't work with a full library type. So it looks as if we might get stuck with a 3 out of 6 implementation that comes with 3 drawbacks.

[IgorStepanov]

Do you have a list of those? I know that ubyte[ushort] = [0: 1, 1: 2]; is problematic, because the compiler uses the left hand side to infer the literal type. In a library one could implement this as AA!(ushort, ubyte, 0, 1, 1, 2)().

The main difference is a multiple-indexing issue:

int[int][int] aa;
aa[1][2]++;

This code can't be implemented in a library type using current opIndex rules.

This pull uses vtables and generates code for each template instance, that's already worse.

Ok, agree with this point, that's little bit worse. (10%)

This will break code which is really worse.

This will not break any correct code. If you tell about @safe/nothrow attributes For example we may read in dmd spec:

Nothrow functions do not throw any exceptions derived from class Exception.

If AA hole would be acceptable, dmd spec would say something like

Nothrow functions do not throw any exceptions derived from class Exception, except associative array features which may scoff the type-system at discretion.

This is a hole which must be closed as soon as possible. If someone abused this hole, he himself to blame.
Otherwise please write the correct code, which will be broken after this PR.

We need to come up with an AA implementation that is much better than the current to justify all the effort.

I think type-system correctness and ctfe-friendliness is sufficient reasons.

see here

This is really breakage the existing code unlike this work:)

These would be my requirements for a good generic AA implementation:

1, 2, 5, 6 are already done (with the new aaLiteral we have removed redundant copying for [Foo(1), Bar(2)]).
3 may be simply implemented with the additional dmd patch (It was be implemented, but you say that dmd patch should be splitted to ctfe and aaLiteral parts)
4 now done for literals. However if we will merge current work, I'll able to gradually replace non-template API functions like aaGetX with template versions. (one per PR)

dmd pull already adds a lot of code that is specific to the vtable implementation but wouldn't work with a full library type.

Only one row: aaLiteralCreate call inside IndexExp::semantic. The aaLiteral code will be similar for library AA, I think. The main advantage of vtable implementation is a avoiding of dmd modifying for it.

******************************************************
My resolution:
If you insist I'll have to agree. However I very dislike this idea.
Adding two similar types to language will create a lot of problems. This will be not fully similar types, because there are at least one thing which can not be implemented in library type (multiple indexing).
Thus, people will need to replace pretty internal aa with ugly template type, but save internal aa in some cases. The next good step is a adding special helper functions for interoperability between library AA and builtin AA. Given how quickly we usually make decisions, it will last at least six months.
After that we will deprecate library AA and will merge builtin AA and library AA.
And all projects which uses new library AA will should rework all code to the new builtin AA.
Descendants will curse us.

The my main argument: this implementation doesn't break any correct existing code. If someone forgot to add nothrow attribute to really nothrow toHash, he can easily do it. Otherwise, if this toHash is not nothrow, this is a user bug and user should fix it. You suggest to add a temporary type which all users should start to use instead builtin AA for some period and should replace it with builtin AA back after this period.
The second argument: this work closes a lot of holes immediately and allows to improve AA work gradually and without shocks.

However, the decision is yours, I'll do as you say.

[IgorStepanov]

@MartinNowak Please, tell your decision about this.

[MartinNowak]

Sorry for being late, it's a really tough decision.

The main difference is a multiple-indexing issue:

int[int][int] aa;
aa[1][2]++;
This code can't be implemented in a library type using current opIndex rules.

Ah right remember that one. Any known clever hacks to implement it?
Seems like opIndex could return a rvalue helper (storing the key and a ref to the aa), that is implicitly convertible to an lvalue or an rvalue, thereby deferring the lookup.
Any other things that are difficult in a library type?

1, 2, 5, 6 are already done (with the new aaLiteral we have removed redundant copying for [Foo(1), Bar(2)]).

Redundant copying yes, but not redundant postblitting.
Might be solvable, see IgorStepanov#1 (comment).

This will not break any correct code.

Correct or not is the wrong metric, "it currently compiles and works but broke with the new release" is the right one. This is a basic type, the fallout of such changes with be immense.

If AA hole would be acceptable, dmd spec would say something like

DMD spec is not the best authority, get's updated every once in while when someone filles a Bugzilla and makes a pull.

This is a hole which must be closed as soon as possible. If someone abused this hole, he himself to blame.

Sorry to say, but that's the wrong attitude. We're trying to become a more professional language and as such cannot afford to flush working production code down the toilet, because of something that someone should have read.

I think type-system correctness and ctfe-friendliness is sufficient reasons.

Both are indeed very good ones.

If someone forgot to add nothrow attribute to really nothrow toHash, he can easily do it.

Not really, because the affected code might be in a different library. Those code breakages start to cascade very badly since we have dub. I was recently baffled by the fact that vibe.d would only build on 2.066.1 (see here).

If you insist I'll have to agree. However I very dislike this idea.

It'd be really unfortunate to have 2 AA types, but it'd be the same with a newAA switch.
In any case we need to come up with a deprecation path for the old broken AA semantics, so I'm open for any idea that works.

The second argument: this work closes a lot of holes immediately and allows to improve AA work gradually and without shocks.

Gradually breaking code isn't better, because we'd have to break things several times.

[MartinNowak]

@MartinNowak Please, tell your decision about this.

I was hoping for a much better AA implementation, but this is merely the same as the old rt.aaA, now even more complicated.

Regarding our recent brainstorming about open addressing.
You had a great idea (though complex) how to make deletion cheaper.
On a second though tombstones are a good (and simple) solution. Because you resize the table when the load factor goes < 0.5, you'll get rid of all dead entries anyhow or they simple get reused during insertion. I also did some research and everybody uses them for deletion with open addressing.

[IgorStepanov]

Sorry to say, but that's the wrong attitude. We're trying to become a more professional language and as such cannot afford to flush working production code down the toilet, because of something that someone should have read.

Just A Moment. Looks like we made noise from scratch. Current dmd part doesn't check attribute correctness. Yes, aaLiteral instance may be impure, throwable or system. However, we doen't create CallExp and don't call semantic for it. We simply get a aaLiteral back-end symbol and manually generate calling code for it. Thus no changes and no breakage now.

Redundant copying yes, but not redundant postblitting.

There was only one redurant postblit call case: when you pass struct literals to aaLiteral.
However, new aaLiteral implementation works similar old AA literal. We manually generates arrays of arguments (inside e2ir) and call postblit if it needed. And we never call dtors for this arrays. Thus we may simply bitwice copy keys and values inside AA constructor. And I do that.
Yes, may be redurant postblit calls for opIndex arguments, but current transitional implementation doesn't use opIndex.

Gradually breaking code isn't better, because we'd have to break things several times.

To sum up: there no breakage for redurant postblit calls and attributes. I don't see another breakage cases. If you see it, please publish it and I'll thing how we can fix it.
Have you another arguments aganist merging of this (transitional) work with vtbl?

Sorry for being late, it's a really tough decision.

Starting from this point, over two months, ping to me will more debt than before. However I don't leave this and will do work when I'll have a time.

I was hoping for a much better AA implementation, but this is merely the same as the old rt.aaA, now even more complicated.

I was hoping for a much better AA implementation, but this is merely the same as the old rt.aaA, now even more complicated.

Unfortunately, even if we implement full library-defined AA, it will be complicated, because we should handle different hard cases like non-copybale keys or values.
However this implementation is simler then old in some cases. For example we don't need to create complex Entry allocate algotithm (including align calculating code) because we know static types of Key and Value. Thus there are no special code for precise GC needed.

[IgorStepanov]

@MartinNowak another one ping :)

[MartinNowak]

Thanks for your effort on this Igor and sorry for realizing do late that it wouldn't work out. Let me close that pull for now, it might have given us enough insight to succeed the next time.

[IgorStepanov]

@MartinNowak
Ok, as I see, aa implementation has been critially changed since the my previous visit.
However, I want to suggest a small enhancement with large practical effect: compile-time constructing of AA literals.
Why does it needed for us?
Static function table are needed for many task with compile-time code generation.
For example, if you want to implemenat routing for a REST server, you may want to use hash-tables, and it may be constructed at compile-time.
Different static dictionaries, error-code tables are also typical task.
What we need to implement this feature?

1. Start to use core.internal.hash.hashOf in TypeInfo.getHash (not hard todo that)
2. Implement template aaLiteral function, which will be used only for CTFE-constructing (thus no breakage of existing code).
3. Add compiler support of aaLiteral for constructing AA in AssocArrayLiteralExp::toDt
   What do you think about it?

[IgorStepanov]

@MartinNowak ping

[MartinNowak]

We were at this point already were we had 2 different AA types in the compiler, it was a mess.

Add compiler support of aaLiteral for constructing AA in AssocArrayLiteralExp::toDt

You can't do that in the toDt phase, because that's too late for semantic analysis.

Static function table are needed for many task with compile-time code generation.

I don't question the value of that, but it likely still adds a lot code/complexity/maintenance for one specific goal.
In the meanwhile I've recycled your vtable idea, see #1282. Let's see whether that'll find enough acceptance.

For example, if you want to implemenat routing for a REST server, you may want to use hash-tables, and it may be constructed at compile-time.

You can use a specialized hash table that supports perfect hashing for that.

[IgorStepanov]

We were at this point already were we had 2 different AA types in the compiler, it was a mess.

There is some code duplication, but it is not two different AA. We will need only two aa literal constructors: one for runtime (existing _d_assocarrayliteralTX) and one for CT (which will follow the _d_assocarrayliteralTX code)

You can't do that in the toDt phase, because that's too late for semantic analysis.

I know, I told simplistically.=) We are partialy done it in my implementation.

You can use a specialized hash table that supports perfect hashing for that.

Of course, I can do it, but the native support of ctfeble AA will be better.

In the meanwhile I've recycled your vtable idea, see #1282. Let's see whether that'll find enough acceptance.

Awesome!
Do you want to insert this type to library without replacing of the runtime AA for testing period?
If it will be ctfeble, there will be additional motivation to use this implementation instead of runtime AA.

[MartinNowak]

Do you want to insert this type to library without replacing of the runtime AA for testing period?

The idea is to provide a much better alternative while slowly pushing people out of the semantic special behavior of the built-in AA. But before that can happen we need AA literals for library types.

[yebblies]

The idea is to provide a much better alternative while slowly pushing people out of the semantic special behavior of the built-in AA. But before that can happen we need AA literals for library types.

Wait what? You've actually given up on matching the builtin AA semantics? Why?

[MartinNowak]

Because of things like ++aa[key1][key2].

[MartinNowak]

If we can add that to the language, fine, but the attribute changes alone require a slow transition.