macro based Modifier for Record

[eugengarkusha]

1)implemented macro based Modifier
2)implemented Updater and Remover in terms of Modifier
3)fixed the following inconsistency in record syntax:
before: "updated" method was used to update and add new fields and it was possible to occasionally add another field with same key(in case of providing wrong value type in updated method).
now : "updated" method only updates the field if the record contains appropriate key and value types, "add" method adds new field if record does not contain the field with same key.

IMPORTANT NOTE:
This change breaks binary compatibility with 2.3.0.

[eugengarkusha]

Working on implementing Select in terms of Modify. It will turn it into single implementation of all CRUD operations

[eugengarkusha]

Introduced Crud typeclass responsible for crud operations on Record. It substitutes Updater, Modifier, Remover and Selector. Selector is not yet removed because Crud marco is not being expanded when spawned inside WitnessWith.

[milessabin]

Thanks!

I don't think it will be possible to merge this as is ... there are many people depending on the current type classes so removing them in this way isn't possible. Alternative implementations of the current type classes are possible (with the proviso that we need to maintain binary compatibility within the 2.3.x series), so maybe you could move back to (some iteration of) what you had earlier for me to review?

[eugengarkusha]

Would it be acceptable to

1. restore Remover, Updater and Modifier(Selector is still there) for backward compatibility. (Mark them as deprecated?).
   2)leave all related operations in syntax.record and lenses Crud-based.

Thanks.

[milessabin]

I suggest making the smallest change that restores the original operations and binary compatibility whilst adding the new functionality you want so that I have something I can start to review.

[eugengarkusha]

Oh, i see that point 2 is not an option. Will be back with updates upon restoring of binary compatibility.

[alexarchambault]

Like in #563, you can keep binary compatibility with non-implicit stubs along the new code.

[eugengarkusha]

Yes looks neat , will go this way!
Thanks

[eugengarkusha]

@alexarchambault , your great advice helped me with JVM binary compatibility, now have you any ideas of how to fix ScalaJs tests?

[eugengarkusha]

blocked by the following ScalaJs issue scala-js/scala-js#2307

[eugengarkusha]

Issue scala-js/scala-js#2307 is fixed by sjrd and should be available in next ScalaJs release(0.69?)

[milessabin]

Good job. I'll bump the Scala.js version as soon as it's available and we can proceed from there.

[eugengarkusha]

sjrd suggested the following quick fix :
"As a work around to immediately fix your issue, you can add @noinline on def unsafeUpdate"
It fixes the problem completely, so we might proceed right away and remove @noinline after next ScalaJs release!

[milessabin]

Can you give us an indication of how this affects compile times? Something along the lines of the timings in #486 and the example in #420.

[eugengarkusha]

All tests were made involving get, updated, updateWIth and remove operations simultaneously:

#420
1)initial example with uncommented fields took ~10 seconds to compile :
2)modified example with twice more fields (100) took ~20 seconds to compile :
2)increasing the amount of operations by 10 times(resulting in 40 operations on 100 fields record) had no significant impact on compile times (+ 2-3 seconds)

#486
1)initial example with no modifications took ˜ 6 seconds to compile
2)modified example with 'k15 is involved in get, updated, updateWIth and remove operations 10 times in a row(get: 4, update: 2, updateWith: 2, remove: 2): took ~9 seconds

[milessabin]

I don't follow. Can you say what 1) and 2) are?

[milessabin]

It would be really helpful to have a couple of examples which are infeasibly slow to compile before this change, but which compile in an acceptable amount of time after it.

[eugengarkusha]

I have edited my previous response, hope it is more clear now.
Example of infeasibly slow compilation before these changes:
1)updateWith operation on Record with 50-70 fields(or many updateWith operations of shorter records)
2)Remove operations were also slower(have not tested the limits)

[eugengarkusha]

Should I provide a repo with examples?

[milessabin]

That would be really helpful :-)

[eugengarkusha]

Here it is :
https://github.com/eugengarkusha/ShapelessPR-567

Currently depends on shapeless 2.3.1-snapshot(locally published branch that contains these changes)
Switch to 2.3.0 to see slow compilation
Thanks!

[milessabin]

Thanks for that! :-)

[eugengarkusha]

YW, pls notify me if you need something else

[milessabin]

I'm good for now. Give me a day or two ...

[eugengarkusha]

Thanks !

[milessabin]

I've tried out the test project and the speedups are great! Thank you!

I would like you change the structure of this change before I merge it though. Rather than having a single type class which does the work for all the operations could you stick with the current structure (ie. one type class for each operation, the existing one for all but the add operation which is new) and have your macro materialize instances of those using the strategy you've adopted here.

The benefit of this is particularly visible wrt update: if the existing Updater type class simply has a new materializer implementation, then the replace syntax is unnecessary and code using update will benefit from your changes as soon as it is recompiled against a new shapeless release.

[eugengarkusha]

Great, thanks for your feedback!

Please correct me if I am wrong:

1)The task is to materialize different type classes(having different method signatures) with a single macro(if it is not the case then would you pls be so kind to provide a bit more formal task definition ).

2)"replace" semantics differs from "updated" as it only supports "replace" operation ("updated" also acts as "add") so the code using "updated" method won't compile in number of places in case of drop-in replacement.

Thanks!

[milessabin]

Apologies about the misunderstanding wrt replace vs. update ... yes, a separate replace method makes sense.

Am I right that you're adding two fundamentally new operations on records: replace (which is like the existing Update except that it will not add extra fields) and add (which adds a field only if it is not already present)?

If that's the case I would prefer to see those two operations added as independent type classes (eg. Replacer and Adder). I understand the motivation behind merging them and the existing Selector, Updater, Remover etc. as Crud, and it might have made sense to do things that way at the outset but, as things are, I think it's better to continue with these as independent type classes.

Each of these type classes requires a separate implicit macro materializer method, but those can delegate to a single shared method which contains the bulk of the implementation.

As well as that, I'd like to see the method unsafeUpdate moved to object HList ... maybe renamed to unsafeUpdateWith to avoid overloading with the existing unsafeUpdate.

You've done all the hard work already, I'm really just proposing shuffling things around a bit ... I hope you don't mind.

[eugengarkusha]

1)Regarding Update and Add: yes you are right.
2)Sure I don't mind, moreover it is an honour for me to be useful for Shapeless.
Thanks for the explanations!

[eugengarkusha]

The requested implementations are provided.
Only thing that bothers me is that instead of "scrapping my boilerplate" I have written more boilerplate.
Is it really important to stick to the existing conventions even if this requires to act against one of the the main goals this library manifests to achieve?
In future releases It could be possible to replace 4 existing typeclasses with one and avoid adding 2 new typeclasses and materializers, still keeping complexity around the same level .

I apologize in advance in case if these questions are inappropriate, I might have overlooked some details which in this case I am asking you to point out.

Thanks!

[milessabin]

Hmm ... that didn't come out quite the way I was expecting.

Why have you added the UnsafeXxxx type classes?

[eugengarkusha]

I may have misunderstood you. Will try to do better today

[eugengarkusha]

Could you pls have a quick glance on this branch again.
Does it look better now?
Pls note that a cast to Aux in each type class materializer is caued by the fact that I dont know how to encode Out type member in quasiquotes(asked in shapeleess and scalamacros gitter channels with no result).

[milessabin]

I would really prefer this to be called unsafeUpdateWith.

[eugengarkusha]

1)got it
2)it expresses an intent of removing the redundant unsafeGet(the same is for unsafeUpdate) method in future releases. Or this also would be not correct approach?

[milessabin]

First, my sincere apologies for taking so long to get back to you on this.

On 2) I think it would be better to go with unsafeGet and express the intent to switch to a different implementation by creating a follow up issue here marked for the next major release.

[eugengarkusha]

Hi!
Just want to confirm, that i have seen your response, thanks!
Totally busy right now, will get back to this asap

[milessabin]

No worries! :-)

[milessabin]

I've just pushed this which is partly inspired by the work you did on this PR ... many thanks for that.

Some of the changes are cosmetic (eg. I don't particularly like the "crud" terminology). There were some technical issues as well. You were creating anonymous subclasses in your macros rather than instantiating a concrete class ... that can result in the creation of many more class files. I also factored your very general crud method into several simpler methods, this makes the use of null unnecessary and eliminates some runtime boxing in the modifier case.

On the additional operations, I added replace but used the Updater type class with Selector used as a constraint to guarantee that the replacement element was of the same type as the element being replaced. And I decided against add ... I didn't feel that an append operation was the right thing to add if that was all that was added, and on the other hand there are too many permutations to add all the plausible variants. So instead I added a LacksKey type class which can be used as evidence in conjunction with any operation where you want to ensure that only new keys are added. The test gives an example of how this might be used.

The main thing is that your compile time benchmark project now compiles in a sensible amount of time ... so thank you very much for encouraging me down this route, and I do hope you don't mind me hijacking this PR.

[eugengarkusha]

Great news!!
I will study the work you've done in this and try to learn a lesson from it.
Thanks!