add #:immutable? keyword argument to functions that return strings, byte-strings, or vectors

[AlexKnauth]

This adds an optional #:immutable? keyword argument to all of the functions from racket/base (that I could think of) that currently return mutable strings, byte-strings, or vectors.

> (immutable? (string-append "abc" "def"))
#f
> (immutable? (string-append "abc" "def" #:immutable? #t))
#t
> (immutable? (number->string 5))
#f
> (immutable? (number->string 5 #:immutable? #t))
#t
> (immutable? (build-vector 4 add1))
#f
> (immutable? (build-vector 4 add1 #:immutable? #t))
#t

This is an alternative to #1219.

[gus-massa]

I was analyzing the problem with the tests. The optimizer can't inline keywords procedures, so it can't see that (verctor? (make-vector 5)) is #t with the new version of make-vector.

A real fix is difficult. It's necessary to force the inlining of the internal procedure make-keyword-procedure, and allow the inlining of struct-procedures. And then, fix a few minnor details, like adding the type of vector->immutable-vector to the optimizer. I guess it's a few months away.

(Perhaps it's possible to fix these issues with some macros. Instead of creating a function make-vector, create a macro that looks for keywords and that transform itself into a function when it's not in application position. But it's a problem is the bind is mutated.)

My easy alternative is to modify the test meanwhile. I wrote a commit that changes make-vector to k:make-vector that is the renamed version of the primitive in the #%kernel. I also made a few similar renaming, so the commit is long, but the relevant lines for this are only a few. It can be merged before or after your commit if necessary.

Commit: gus-massa@5a9a467 ( travis ) (look near optimize.rktl#L1359 )

By the way, you forgot struct->vector .

[AlexKnauth]

Oh, okay. Would it make sense to make it a non-keyword argument somehow? But then what about functions like string-append that take rest arguments?

[gus-massa]

I think it's better to keep your version with the keywords because it looks better for usability, but I don't know the opinion of the rest.

I hope it's possible to fix the optimization details later.

As a proof of concept, I wrote a version of make-vector that can be (almost) reduced by the current optimizer. The code is horrible, I should have used syntax-protect and syntax/loc, ...
And another side effect is that this is backward incompatible because make-vector is a macro that transforms into a function, not a real function.

#lang racket
(require (only-in '#%kernel (make-vector k:make-vector)))

(define (make-vector/noval/nokeyword len immutable?)
  (if immutable?
    (vector->immutable-vector (k:make-vector len))
    (k:make-vector len)))

(define (make-vector/nokeyword len val immutable?)
  (if immutable?
    (vector->immutable-vector (k:make-vector len val))
    (k:make-vector len val)))

(define-syntax-rule (define/rename name (rename args ...) body ...)
  (define name
    (let ([rename (lambda (args ...) body ...)])
      rename))) 

(define/rename make-vector/proc (make-vector len (val 0) #:immutable? immutable?)
  (if immutable?
    (vector->immutable-vector (k:make-vector len val))
    (k:make-vector len val)))

(define-syntax make-vector
  (make-set!-transformer
   (lambda (stx)
     (syntax-case stx ()
       [(_ len)
        #'(k:make-vector len)]
       [(_ len val)
        #'(k:make-vector len val)]
       [(_ len kw:immutable? immutable?)
        (eq? '#:immutable? (syntax-e #'kw:immutable?))
        #'(make-vector/noval/nokeyword len immutable?)]
       [(_ kw:immutable? immutable? len)
        (eq? '#:immutable? (syntax-e #'kw:immutable?))
        #'(make-vector/noval/nokeyword len immutable?)]
       [(_ len val kw:immutable? immutable?)
        (eq? '#:immutable? (syntax-e #'kw:immutable?))
        #'(make-vector/nokeyword len val immutable?)]
       [(_ len kw:immutable? immutable? val)
        (eq? '#:immutable? (syntax-e #'kw:immutable?))
        #'(make-vector/nokeyword len val immutable?)]
       [(_ kw:immutable? immutable? len val)
        (eq? '#:immutable? (syntax-e #'kw:immutable?))
        #'(make-vector/nokeyword len val immutable?)]
       [(_ . any) #'(make-vector/proc .  any)]
       [_  #'make-vector/proc]))))

(vector? (make-vector 5))
;==optimized==> #t

(vector? (make-vector 5 #:immutable? #f))
;==optimized==> #t

(vector? (make-vector 5 #:immutable? #t))
;==optimized==> (vector? (vector->immutable-vector (make-vector 5)))

[AlexKnauth]

But this wouldn't work if I made the optional argument default to the value of a (create-immutable-vector?) parameter, would it? Unless the optimizer could know the value of that parameter, it wouldn't be able to know which branch it was.

[gus-massa]

I totally missed the part about parameters. In my example, it can be fixed with something like

[(_ len val)
 #'(make-vector/nokeyword len val (create-inmmutable-vector?))]

Now the reductions are not perfect, but make-vector/nokeyword can be inlined and with some work in the future they can be optimizad away. The difficult part is optimizing the keyword machinery.

IMHO, it's a bad idea to use parameters in a function like make-vector because it's too low level. I'm slightly worried about the performance, but I also don't like that parameters are too similar to a global flag. (I don't have this objection for high level functions like display, because they do a lot of crazy stuff and have a lot of options, so it's impossible to write all the details as explicit arguments.)

In this case, I'd like to have a fixed default. I'd like to make the immutable version the default, but it's backwards incompatible, so I think this change should be delayed to Racket2. So, I think that the next best option is to add the keyword but keep the mutable version as the default. This can make some calculations slower, but the difference can be fixed.

Another comment about my horrible code: It has a few wrong corner cases, for example

(void (make-vector #:immutable (display "X") 7 (display "Y"))) 
; ==show==> YX instead of XY

[AlexKnauth]

Ok, so for lower-level functions like make-vector to avoid the parameter, does it make sense to have immutable versions be provided by a racket/immutable library like in my other pull request? (#1219)

[gus-massa]

I found a small problem in your implementation with the message of the error when a function has an implicit argument, for example in:

(make-vector -1)

The error is

make-vector: contract violation
  expected: exact-nonnegative-integer?
  given: -1
  argument position: 1st
  other arguments...:
   0
  context...:
    [...]

Instead of

make-vector: contract violation
  expected: exact-nonnegative-integer?
  given: -1
  context...:
    [...]

I was looking at the implementation of keyword procedures. kw.rkt#L1026

The implementation binds the identifier in the definition to a macro that expands to a hidden procedure. So I guess it's possible to change the implementation of all the keyword procedures to something like the macro that I wrote before (fixing all the details, corner cases and errors :)). But it's out of scope of your commit, perhaps I should send a feature request to the mailing list, to try to convince someone that knows the details to fix it.

I'd like that in some code like

(define (f x #:y (y 0)))
  (list x y))

At least (f 5) is expanded to a direct application of a (hidden) function, not a call to checked-procedure-check-and-extract. So it's adding an unused keyword to a procedure doesn't prevent the optimizations and reductions, and probably doesn't increase the run time.

Extra points for expanding (f 5 #:y 3) to a direct application of a function. This looks more difficult, because the keywords can be reordered (see my previous comment).

I made a few small tests, and apparently the problem is only with optional keywords, all the other cases seams to be already written in the more efficient way. :)

[AlexKnauth]

No, I don't think this has anything to do with keyword arguments. It's because I'm doing the equivalent of this:

(define (my-make-vector n [v 0])
  (make-vector n v))
(my-make-vector -1)

And that means that the core make-vector is actually getting two arguments.

[gus-massa]

I agree that there are two different problems.

One is that the optional keyword arguments are expanded in a way that is not friendy to the optimizer. This has to be fixed elsewhere.

The other problem is that after using the automatic optional argument, the distintion between a missing argument and one that coincides dissapears, so in case of an error, the end user can get confused by the ghost argument.

I was thinking in something like:

(define missing-argument (gensym 'missing))

(define (my-make-vector len (val missing-argument))
  (if (eq? val missing-argument)
    (make-vector len)
    (make-vector len val)))

(my-make-vector -1)

(It's not 100% optimizer friendly, but it raise the same errors that the make-vector in the kernel.)