Within-module eval can break precompile

[tecosaur]

This seems like it might be necessary to avoid precompile problems otherwise seen in JuliaLang/julia#51811?

Thoughts on this would be welcome.

[caleb-allen]

Very nice, I was quite literally on my way to this repo to address this exact issue.

This allows me to declare a top level const within REPL.jl, changing

const JULIA_PROMPT = "julia> "

to

const JULIA_PROMPT = styled"{repl_prompt_julia:julia> }"

Out of curiosity, is this interpolation mechanism similar to the way julia strings operate?

[tecosaur]

Glad this helped. I've been chatting with Kristoffer and I'm going to probably change the approach here in a sec to avoid conditioning on Base.generating_code() and then merge this.

Out of curiosity, is this interpolation mechanism similar to the way julia strings operate?

What's "this" interpolation, I can't see any in the JULIA_PROMPT string?

[caleb-allen]

What's "this" interpolation

My apologies for the ambiguity. To clarify, my question regards the way in which styled_str interpolates and evaluates inner-julia code, rather than a specific question about my example.

[tecosaur]

Ah right, cool. Just wanted to make sure I answered the right question 🙂.

I think just looking at the macroexpansion perhaps could help with this? To give a few simplified examples:

julia> @macroexpand styled"hello $you"
:(annotatedstring("hello ", you))

So in this case, this should be pretty much equivalent to string("hello ", you).

If we want to style an interpolated value, it's a little more complex.

julia> @macroexpand styled"hello {bold:$you}"
:(annotatedstring("hello ", let str=string(you); AnnotatedString(str, [(1:ncodeunits(you), :face => :bold]) end))

The complication here is that we need to know how large you is to determine the correct annotation width, hence the let.

The other way we can do interpolation is in the face/annotation itself, which looks like this:

julia> @macroexpand styled"hello {$myface:you}"
:(annotatedstring(AnnotatedString("hello you", [(7:9, :face => myface)])))

In this case, since the content is static hello and you are combined, and we know the annotation region, the only dynamic part of this is myface.

Hopefully that clears things up, if not I'm happy to answer any follow-up questions 🙂.

[tecosaur]

Ok, so this PR makes it possible to precompile this stdlib into system image, which is eminently desirable. We may want to change the particular approach it takes from manually applied hygiene to interpolating every function/type of the eval'd expressions (though I'm worried this could make it easier to accidentally introduce subtle bugs).

So, while perhaps not the final approach, this PR represents an improvement to the current situation, and so I'm going to go ahead and merge it.

[caleb-allen]

Thank you for your thoughtful answer.

I'm curious, is the implementation side similar? I haven't fully grokked the styled macro at this point, but my naive understanding is that it's taking a racket-esque approach, by which I mean implementing functionality like $ interpolation via macro (operating on the string as a list of expressions, as opposed to, say, evaluating interpolated expressions prior to the construction of a string as a stage in parsing or compilation). To that end, how does Julia interpolate strings? And should we expect that these eval-related interpolation issues are unique to StyledStrings because of its macro based approach?

I suppose this discussion may be beside the point of pre-compilation, but I can't help but be curious since I don't have enough familiarity with Julia's internals to have an understanding of how a string goes from, let's say, a string literal in a file, e.g. "Julia Version $VERSION" to a string in memory with the value "Julia Version 1.10.0-beta3". My guess is that such interpolation is built in to Julia's string related syntax rules, so that constructing a string literal evaluates all interpolated values, convert them to a string via string, and then appending them all together. And my guess is that this would operate at an earlier stage of compilation than macro evaluation? And therefore that emulating such a feature requires a macro-based approach? I'm sure it's a matter of searching and reading Julia's code to understand it all. Just thinking out loud.

As a side note, I hope my intention comes across correctly. I'm naive about these inner workings and I'm trying to understand them, not critique this approach; I very much appreciate the strengths of macro-based "languages-within-a-language" solutions (and would be thrilled to see initiatives to make hygienic macros easier to write in Julia).

[tecosaur]

I'm curious, is the implementation side similar?

I'm not fully aware of the internals of Julia string interpolation, but from my understanding — not at all. Julia string interpolation is handled at a parser level, and IIRC ends up being equivalent to a string(...) form.

By contrast, here the string macro is given the already parsed string (just without interpreting \-escape sequences), and the parser-level interpolation is manually emulated during the expansion of the styled string macro.

From your thoughts, it sounds like your speculation and my current understanding of the situation mesh pretty cleanly?

And should we expect that these eval-related interpolation issues are unique to StyledStrings because of its macro based approach?

The problem here is that styled"" gets extra fancy and tries to evaluate expressions that it knows are valid targets for constant-eval, but the compiler doesn't, and this allows for a redundant-annotation-merging pass to be applied at compile time in some cases.

As far as I'm aware, there aren't any other macros currently trying something similar.