racket-hash-lang-mode not highlighting buffer with font-lock

[dpk]

I’m trying out the head of the hash-lang branch on Emacs 28.1 with Racket 8.6. (I know it’s pre-release, so you might already know about this problem.) It works and the buffers are definitely talking to the backend, which is great! (Among other things, Scribble indentation works correctly like in DrRacket.) But neither font-lock nor recognizing when the #lang has changed are working. Specifically, I can colour the whole buffer by M-: evaluating (racket--hash-lang-font-lock-fontify-region (point-min) (point-max)), which works fine, but it doesn’t automatically font-lock the buffer as I type, and if I start an empty buffer and add any #lang line, it doesn’t recognize the new language I set.

after-change-functions in a racket-hash-lang-mode buffer are (t racket--hash-lang-after-change-hook), and font-lock-fontify-region-function is indeed racket--hash-lang-font-lock-fontify-region, so I don’t understand what’s up here.

[greghendershott]

Hi! Thank you for trying this.

I'll take a look. It worked for me, last I tried... but I haven't tried in awhile.

(If it still does work for me, I'll follow up with some questions so we can figure it out.)

---

Some background: I've spent quite a lot of time working on this branch, so far -- but not for some months. I think it's in pretty OK shape, but I've been reluctant to commit to merging it. I'm just worried about bugs and performance and maintaining it long-term. Basically, I'm kind of waiting for a "sponsor" -- not financially, but someone who truly needs/wants to use it for real work.

TL;DR I'm OK with it turning out to be much more work, provided it's solving a real problem for someone. 😄

If you turn out to be that person (or the first such person), that's great and I can spend more time on this. (OTOH if you're not willing or able to, that's also OK and I'll probably just keep waiting.)

[greghendershott]

It's still working for me, using Emacs 25.2 and Racket 8.6.0.8.

In the following little example file, I make various changes. The appearance changes as expected within a second:

• delete the at-exp meta lang
• insert it back
• change the lang to scribble/text
• change it back to at-exp racket
• delete brand new text e.g. something to be colored as a "string"
• indent inside the at-exp curly braces, as well as the "normal" expressions e.g. if

#lang at-exp racket/base

#;
(let ([x 12])
  (let ([x x])
    (cond [#t
 (x 12)])
    (if 1
        2
        3)))

(require racket/format
         "scheme.rkt")
@~a{if this were text
    I would just keep typing.}
@~a{this is just text
    this is more just text.}
(if 1
    2
    3)
(if 2
    @~a{foo bar
        or do something else
        blah blah blah}
    3)
;; comment
(if 1
    (or "string" 23)
    (cond [(equal? 1 2) #t]
          [else #f]))
(if 1
    3 ; )
    4)
;; asdfadsf
123
"123 (asdf) adsfafd" 1234 ;;#:keyword
#<<HERE
this is a here string
HERE
(if 1
    2
    4)
(if 1
    3
    (cond
      [#t 1]
      [else 3]))
(module+ test
  "hello I am the test submod")

---

I can try Emacs latest built from source, as well as refresh my Racket built from source.

[greghendershott]

• As well as Emacs 25.2, it also works for me with Emacs 29.0.50 (which I built from source around September 1). So this seems unrelated to you using Emacs 28.2.

• As well as Racket 8.6.0.6 (build of WIP 8.7), it also works for me with Racket 8.6 release.

• Could you please M-x racket-bug-report and paste here for me? Probably it's related to something else you're using in Emacs. I could probably help suggest things for you to try disabling (but once I know what it is, of course I'd resolve whatever the incompatibility is).

[greghendershott]

Another thought: It sounds like racket--hash-lang-on-notify isn't being called (and in turn isn't calling racket--hash-lang-on-new-lang and racket--hash-lang-on-changed-tokens).

As a guess, this could happen if the Emacs Lisp code in racket-cmd.el wasn't re-evaluated or re-compiled after switching git branches?

So as a sanity check I'd recommend deleting the Racket Mode .elc files, and, completely restarting Emacs. After that, does the problem persist for you?

(If it still persists, then I'd like to ponder your racket-bug-report details.)

[dpk]

Thanks for the extensive feedback!

Some background: I've spent quite a lot of time working on this branch, so far -- but not for some months. I think it's in pretty OK shape, but I've been reluctant to commit to merging it. I'm just worried about bugs and performance and maintaining it long-term. Basically, I'm kind of waiting for a "sponsor" -- not financially, but someone who truly needs/wants to use it for real work.

TL;DR I'm OK with it turning out to be much more work, provided it's solving a real problem for someone. 😄

If you turn out to be that person (or the first such person), that's great and I can spend more time on this. (OTOH if you're not willing or able to, that's also OK and I'll probably just keep waiting.)

I would love to see hash-lang-mode released, but to be honest I am probably not the right person to be a sponsor. I’m mostly in the RnRS Scheme world and only dabble in Racket occasionally.

Another thought: It sounds like racket--hash-lang-on-notify isn't being called (and in turn isn't calling racket--hash-lang-on-new-lang and racket--hash-lang-on-changed-tokens).

As a guess, this could happen if the Emacs Lisp code in racket-cmd.el wasn't re-evaluated or re-compiled after switching git branches?

I installed the whole Git branch from scratch, and I don’t see any issue when I do e.g. describe-function racket-start-back-end.

---

My debug information is here. (I upgraded to 28.2 today to see if that would fix the issue, but it didn’t.)

((alist-get 'racket-mode package-alist))
((emacs-version "28.2")
 (system-type darwin)
 (x-gtk-use-system-tooltips UNDEFINED)
 (major-mode racket-mode)
 (racket--el-source-dir "/Users/dpk/.emacs.d/racket-mode/")
 (racket--rkt-source-dir "/Users/dpk/.emacs.d/racket-mode/racket/")
 (racket-program "racket")
 (racket-command-timeout 10)
 (racket-path-from-emacs-to-racket-function UNDEFINED)
 (racket-path-from-racket-to-emacs-function UNDEFINED)
 (racket-browse-url-function racket-browse-url-using-temporary-file)
 (racket-documentation-search-location "https://docs.racket-lang.org/search/index.html?q=%s")
 (racket-xp-after-change-refresh-delay 1)
 (racket-xp-mode-lighter
  (:eval
   (racket--xp-mode-lighter)))
 (racket-xp-highlight-unused-regexp "^[^_]")
 (racket-repl-buffer-name-function nil)
 (racket-submodules-to-run
  ((test)
   (main)))
 (racket-memory-limit 2048)
 (racket-error-context medium)
 (racket-repl-history-directory "~/.emacs.d/racket-mode/")
 (racket-history-filter-regexp "\\`\\s *\\'")
 (racket-images-inline t)
 (racket-imagemagick-props nil)
 (racket-images-keep-last 100)
 (racket-images-system-viewer "open")
 (racket-pretty-print t)
 (racket-use-repl-submit-predicate nil)
 (racket-pretty-print t)
 (racket-indent-curly-as-sequence t)
 (racket-indent-sequence-depth 0)
 (racket-pretty-lambda nil)
 (racket-smart-open-bracket-enable nil)
 (racket-module-forms "\\s(\\(?:module[*+]?\\|library\\)")
 (racket-logger-config
  ((cm-accomplice . warning)
   (GC . info)
   (module-prefetch . warning)
   (optimizer . info)
   (racket/contract . error)
   (racket-mode-debugger . info)
   (sequence-specialization . info)
   (* . fatal)))
 (racket-show-functions
  (racket-show-pseudo-tooltip)))
(enabled-minor-modes
 (auto-composition-mode)
 (auto-compression-mode)
 (auto-encryption-mode)
 (auto-fill-mode)
 (auto-save-mode)
 (blink-cursor-mode)
 (column-number-mode)
 (delete-selection-mode)
 (display-line-numbers-mode)
 (file-name-shadow-mode)
 (font-lock-mode)
 (global-display-line-numbers-mode)
 (global-eldoc-mode)
 (global-font-lock-mode)
 (global-git-commit-mode)
 (global-undo-tree-mode)
 (hs-minor-mode)
 (line-number-mode)
 (magit-auto-revert-mode)
 (menu-bar-mode)
 (mouse-wheel-mode)
 (pcre-mode)
 (racket-hash-lang-mode)
 (semantic-minor-modes-format)
 (shell-dirtrack-mode)
 (show-paren-mode)
 (tooltip-mode)
 (transient-mark-mode)
 (undo-tree-mode)
 (which-key-mode)
 (xterm-mouse-mode))
(disabled-minor-modes
 (abbrev-mode)
 (auto-fill-function)
 (auto-revert-mode)
 (auto-revert-tail-mode)
 (auto-save-visited-mode)
 (buffer-face-mode)
 (buffer-read-only)
 (button-mode)
 (cl-old-struct-compat-mode)
 (compilation-minor-mode)
 (compilation-shell-minor-mode)
 (completion-in-region-mode)
 (context-menu-mode)
 (dash-fontify-mode)
 (defining-kbd-macro)
 (diff-auto-refine-mode)
 (diff-minor-mode)
 (dired-hide-details-mode)
 (eldoc-mode)
 (electric-indent-mode)
 (electric-layout-mode)
 (electric-quote-mode)
 (geiser-autodoc-mode)
 (geiser-mode)
 (geiser-smart-tab-mode)
 (git-commit-mode)
 (global-auto-revert-mode)
 (global-dash-fontify-mode)
 (global-prettify-symbols-mode)
 (global-semantic-highlight-edits-mode)
 (global-semantic-highlight-func-mode)
 (global-semantic-show-parser-state-mode)
 (global-semantic-show-unmatched-syntax-mode)
 (global-semantic-stickyfunc-mode)
 (global-visual-line-mode)
 (horizontal-scroll-bar-mode)
 (ido-everywhere)
 (indent-tabs-mode)
 (isearch-mode)
 (jit-lock-debug-mode)
 (lock-file-mode)
 (magit-blame-mode)
 (magit-blame-read-only-mode)
 (magit-blob-mode)
 (magit-wip-after-apply-mode)
 (magit-wip-after-save-local-mode)
 (magit-wip-after-save-mode)
 (magit-wip-before-change-mode)
 (magit-wip-initial-backup-mode)
 (magit-wip-mode)
 (mail-abbrevs-mode)
 (mml-mode)
 (next-error-follow-minor-mode)
 (outline-minor-mode)
 (overwrite-mode)
 (paragraph-indent-minor-mode)
 (poly-psgml-html-mode)
 (polymode-minor-mode)
 (prettify-symbols-mode)
 (racket-smart-open-bracket-mode)
 (racket-xp-mode)
 (rich-minority-mode)
 (rxt--read-pcre-mode)
 (rxt-global-mode)
 (rxt-mode)
 (semantic-highlight-edits-mode)
 (semantic-highlight-func-mode)
 (semantic-mode)
 (semantic-show-parser-state-mode)
 (semantic-show-unmatched-syntax-mode)
 (semantic-stickyfunc-mode)
 (server-mode)
 (sgml-electric-tag-pair-mode)
 (sh-electric-here-document-mode)
 (shell-command-with-editor-mode)
 (size-indication-mode)
 (smerge-mode)
 (tab-bar-history-mode)
 (tab-bar-mode)
 (temp-buffer-resize-mode)
 (text-scale-mode)
 (tool-bar-mode)
 (transient-resume-mode)
 (undo-tree-visualizer-selection-mode)
 (unify-8859-on-decoding-mode)
 (unify-8859-on-encoding-mode)
 (url-handler-mode)
 (use-hard-newlines)
 (view-mode)
 (visible-mode)
 (visual-line-mode)
 (which-function-mode)
 (window-divider-mode)
 (with-editor-mode)
 (xref-etags-mode))

[dpk]

Just to confirm: deleting the .elc files did not help.

[dpk]

And the variable racket--hash-lang-id doesn’t change when I change the #lang line in the example file in the ways you suggest. It does pick up new hash-langs when a completely new file is opened, though: if I open a new file which already has #lang scribble/text, I can see that one buffer has the value 1 and the other 2. This makes sense because both files also get correct indentation behaviour (one of the few visible effects of different #langs since colouring isn’t affected).

[greghendershott]

Thanks for the responses!

1. Your enabled-minor-modes is quite short/simple/clean. Dammit. 😄 Nothing seems likely to contribute to this; nothing I'd suggest disabling as an experiment.

2. I wouldn't expect the racket--hash-lang-id value to change while you are simply editing the file. It's just an integer identifier that should match the lifetime of racket-hash-lang-mode in a specific buffer. It shouldn't change for a buffer (unless maybe you visit-file and effectively recreate the buffer and re-enable racket-hash-lang-mode for it).

3. However the racket--hash-lang-generation variable will, in each buffer, increment for each edit. I imagine you'll see that if you look: It sounds like the after-change-hook function is being called, and incrementing this, and telling the back end about the changes.

   Instead it seems like the back end is not notifying the front end about lang or tokenization changes -- or if it is, the front end's work to refresh the coloring is having no effect. Exactly what's going on here, I don't know yet.

One thing you could try would be to un-comment the message calls near the start of these two functions -- and of course M-x eval-defun each one to refresh:

(defun racket--hash-lang-on-new-lang (plist)
  "We get this whenever any #lang supplied attributes have changed.

We do /not/ get notified when a new lang uses exactly the same
attributes as the old one. For example changing from #lang racket
to #lang racket/base will /not/ notify us, because none of the
lang's attributes that care about have changed."
-  ;;;(message "racket--hash-lang-on-new-lang %s" plist)
+  (message "racket--hash-lang-on-new-lang %s" plist)
  (with-silent-modifications

(defun racket--hash-lang-on-changed-tokens (_gen beg end)
  "The back end has processed a change that resulted in new tokens.

All we do here is mark the span as not fontified, then let the
usual font-lock machinery do its thing if/when this span ever
becomes visible."
-  ;;;(message "racket--hash-lang-on-changed-tokens %s %s %s" _gen beg end)
+  (message "racket--hash-lang-on-changed-tokens %s %s %s" _gen beg end)
  (with-silent-modifications
    (save-restriction
      (widen)
      (put-text-property beg
                         (min end (point-max))
                         'fontified nil))))

Then, if you make some edits, and look in the *Messages* buffer, do you see any output?

It might help me to know whether if these are being called but for some reason their work isn't having any visible effect, --or--, not even being called in the first place.

[dpk]

Yes, racket--hash-lang-generation does change, and I do get the notifications (example from messing about in a test buffer).

racket--hash-lang-on-changed-tokens 6 180 182
racket--hash-lang-on-changed-tokens 7 180 183
racket--hash-lang-on-changed-tokens 8 180 182
racket--hash-lang-on-changed-tokens 9 180 181
racket--hash-lang-on-new-lang (racket-grouping t range-indenter nil submit-predicate nil)
racket--hash-lang-on-changed-tokens 10 1 182
racket--hash-lang-on-changed-tokens 11 1 8
racket--hash-lang-on-changed-tokens 12 1 9
racket--hash-lang-on-changed-tokens 13 1 10
racket--hash-lang-on-changed-tokens 14 1 11
racket--hash-lang-on-changed-tokens 15 1 12
racket--hash-lang-on-new-lang (racket-grouping t range-indenter nil submit-predicate nil)
racket--hash-lang-on-changed-tokens 16 1 188
racket--hash-lang-on-new-lang (racket-grouping t range-indenter nil submit-predicate nil)
racket--hash-lang-on-changed-tokens 17 1 184
racket--hash-lang-on-changed-tokens 18 1 14
racket--hash-lang-on-changed-tokens 19 1 15
racket--hash-lang-on-changed-tokens 20 1 16
racket--hash-lang-on-changed-tokens 21 1 17
racket--hash-lang-on-changed-tokens 22 1 16
racket--hash-lang-on-changed-tokens 23 1 17
Auto-saving...done
racket--hash-lang-on-changed-tokens 24 1 18
racket--hash-lang-on-changed-tokens 25 1 19
racket--hash-lang-on-new-lang (racket-grouping t range-indenter nil submit-predicate nil)
racket--hash-lang-on-changed-tokens 26 1 181
racket--hash-lang-on-new-lang (racket-grouping t range-indenter nil submit-predicate nil)
racket--hash-lang-on-changed-tokens 27 1 182
racket--hash-lang-on-changed-tokens 28 1 22
racket--hash-lang-on-changed-tokens 29 1 23
racket--hash-lang-on-changed-tokens 30 1 24
racket--hash-lang-on-new-lang (racket-grouping t range-indenter nil submit-predicate nil)
racket--hash-lang-on-changed-tokens 31 1 186

On testing it again, the #lang change is picked up: when I start with a buffer with #lang racket/base and then switch to #lang scribble/base and reindent, it does reindent according to the rules for Scribble. So it’s just a problem with font-lock, in fact.

[dpk]

I uncommented the similar log line in racket--hash-lang-font-lock-fontify-region and indeed, it logs no messages. So this looks like a (somewhat confounding) problem with font-lock, because as mentioned, my font-lock-fontify-region-function is indeed set to racket--hash-lang-font-lock-fontify-region.

[dpk]

Potentially related, potentially an irrelevant other issue: I turned on font-lock-verbose and ran font-lock-fontify-buffer manually. It came up with a load of messages:

Fontifying racket-hash-lang-test2...
fontify-region 1 31
Invalid face reference: parenthesis [24 times]
funcall-interactively: Beginning of bufferInvalid face reference: parenthesis
Invalid face reference: parenthesis [11 times]
funcall-interactively: Beginning of bufferInvalid face reference: parenthesis
Invalid face reference: parenthesis [85 times]

(There are not 85 parentheses in the file. Indeed, there are not that many characters in this test file.)

[greghendershott]

Thanks! That all seems like super helpful information. I'll mull it over, and try some things. Ideally I want to reproduce it here, too, so I can know what to fix, and how, and in fact know I have fixed it.

(I'm pretty sure the "invalid face reference" stuff is a red herring, but I'll make a change to avoid that. (When using e.g. the paren-face package, which defines a parenthesis face, it's nice to use that face. When not, it's harmless in my experience to specify a non-existing face; default will be used. But even so, those font-lock warnings are super noisy.) I temporarily hacked it to use instead of 'parenthesis 'I-DO-NOT-EXIST, and I do get those warnings, but I don't get the bug you're seeing.)

[greghendershott]

What values do you have for the customization variables jit-lock-contextually and jit-lock-context-time?

[dpk]

jit-lock-contextually is 'syntax-driven and jit-lock-context-time is 0.5

[greghendershott]

To the hash-lang branch I just pushed a couple commits:

• commit 3e0f9c7 only uses the parenthesis face when it exists (the warnings you saw)
• commit a42417d avoids some copypasta and improves a couple doc strings

Neither of these should be substantive changes that would fix the problem you're having, though.

I'm very confused. I've been re-reading the doc and source for font-lock and jit-lock, and reloading my brain with some details. I have no mental model, yet, for what you're seeing.

Probably I should set this aside and let my subconscious work on it for awhile. Also I may try with Emacs 28.2 (like you) on the (unlikely?) chance something was broken/different then, but not in 25.2 or 29.x.

[greghendershott]

If you're willing/able to try this branch again, you might get a different result now?

I've still not been able to reproduce this. I tried again recently with more Emacs versions (28.2 and 29.1), and on more platforms (macosx as well as linux).

I really want to merge this branch soon; it's been open for years now. I've worked on it a lot the last month or two; I've started to dog-food it for some real editing of .rkt and .scrbl files. My only hesitation is this issue as well as #669, where something is happening that I can't reproduce, which means I'm overlooking something. 😞

[LiberalArtist]

Following up here from the Discourse thread: this does sound like the issue I'm experiencing!

• They mentioned that indent does work for them. It would be interesting to know for you, too.

Indentation seems to work from my limited testing so far.

• If you move someplace you expect color -- say within a string value -- and press C-u C-x = it will open a buffer with properties about the char at point. If you could copy that for me to see, that would be useful.

Here you go!

             position: 297 of 390 (76%), column: 29
            character: o (displayed as o) (codepoint 111, #o157, #x6f)
              charset: ascii (ASCII (ISO646 IRV))
code point in charset: 0x6F
               script: latin
               syntax: w 	which means: word
             category: .:Base, L:Strong L2R, a:ASCII, l:Latin, r:Roman
             to input: type "C-x 8 RET 6f" or "C-x 8 RET LATIN SMALL LETTER O"
          buffer code: #x6F
            file code: #x6F (encoded by coding system utf-8-unix)
              display: by this font (glyph code):
    ftcrhb:-GOOG-Noto Sans Mono-regular-normal-normal-*-15-*-*-*-*-0-iso10646-1 (#x52)

Character code properties: customize what to show
  name: LATIN SMALL LETTER O
  general-category: Ll (Letter, Lowercase)
  decomposition: (111) ('o')

There are text properties here:
  fontified            t

And here's what I get from M-x racket-bug-report:

((alist-get 'racket-mode package-alist)
 #s(package-desc racket-mode
                 (1)
                 "Racket editing, REPL, and more"
                 ((emacs
                   (25 1)))
                 nil nil "/gnu/store/hga14hb1lg1pnw255n96mwgxd2cy2b3d-emacs-racket-mode-0.0.2-8.65ebeb3/share/emacs/site-lisp/racket-mode-0.0.2-8.65ebeb3"
                 ((:url . "https://www.racket-mode.com/")
                  (:authors
                   ("Greg Hendershott" . "racket-mode-author@greghendershott.com")))
                 nil)
 #s(package-desc racket-mode
                 (1)
                 "Racket editing, REPL, and more"
                 ((emacs
                   (25 1)))
                 nil nil "/gnu/store/hga14hb1lg1pnw255n96mwgxd2cy2b3d-emacs-racket-mode-0.0.2-8.65ebeb3/share/emacs/site-lisp/racket-mode-0.0.2-8.65ebeb3"
                 ((:url . "https://www.racket-mode.com/")
                  (:authors
                   ("Greg Hendershott" . "racket-mode-author@greghendershott.com")))
                 nil))
((emacs-version "29.1")
 (system-type gnu/linux)
 (x-gtk-use-system-tooltips t)
 (major-mode help-mode)
 (racket--el-source-dir "/gnu/store/hga14hb1lg1pnw255n96mwgxd2cy2b3d-emacs-racket-mode-0.0.2-8.65ebeb3/share/emacs/site-lisp/racket-mode-0.0.2-8.65ebeb3/")
 (racket--rkt-source-dir "/gnu/store/hga14hb1lg1pnw255n96mwgxd2cy2b3d-emacs-racket-mode-0.0.2-8.65ebeb3/share/emacs/site-lisp/racket-mode-0.0.2-8.65ebeb3/racket/")
 (racket-program "racket")
 (racket-command-timeout 10)
 (racket-path-from-emacs-to-racket-function UNDEFINED)
 (racket-path-from-racket-to-emacs-function UNDEFINED)
 (racket-browse-url-function racket-browse-url-using-temporary-file)
 (racket-documentation-search-location "https://docs.racket-lang.org/search/index.html?q=%s")
 (racket-xp-after-change-refresh-delay 1)
 (racket-xp-mode-lighter
  (:eval
   (racket--xp-mode-lighter)))
 (racket-xp-highlight-unused-regexp "^[^_]")
 (racket-repl-buffer-name-function nil)
 (racket-submodules-to-run
  ((test)
   (main)))
 (racket-memory-limit 2048)
 (racket-error-context medium)
 (racket-repl-history-directory "/home/philip/.emacs.d/var/racket-mode/repl-history/")
 (racket-history-filter-regexp "\\`\\s *\\'")
 (racket-images-inline t)
 (racket-imagemagick-props nil)
 (racket-images-keep-last 100)
 (racket-images-system-viewer "display")
 (racket-pretty-print t)
 (racket-use-repl-submit-predicate nil)
 (racket-pretty-print t)
 (racket-indent-curly-as-sequence t)
 (racket-indent-sequence-depth 0)
 (racket-pretty-lambda nil)
 (racket-smart-open-bracket-enable nil)
 (racket-module-forms "\\s(\\(?:module[*+]?\\|library\\)")
 (racket-logger-config
  ((cm-accomplice . warning)
   (GC . info)
   (module-prefetch . warning)
   (optimizer . info)
   (racket/contract . error)
   (racket-mode-debugger . info)
   (sequence-specialization . info)
   (* . fatal)))
 (racket-show-functions
  (racket-show-pseudo-tooltip)))
(enabled-minor-modes
 (auto-composition-mode)
 (auto-compression-mode)
 (auto-encryption-mode)
 (auto-fill-mode)
 (auto-save-mode)
 (blink-cursor-mode)
 (buffer-read-only)
 (delete-selection-mode)
 (electric-indent-mode)
 (file-name-shadow-mode)
 (font-lock-mode)
 (global-eldoc-mode)
 (global-font-lock-mode)
 (global-git-commit-mode)
 (global-undo-tree-mode)
 (isearch-fold-quotes-mode)
 (line-number-mode)
 (magit-auto-revert-mode)
 (menu-bar-mode)
 (mouse-wheel-mode)
 (override-global-mode)
 (semantic-minor-modes-format)
 (shell-dirtrack-mode)
 (show-paren-mode)
 (tool-bar-mode)
 (tooltip-mode)
 (transient-mark-mode)
 (undo-tree-mode))
(disabled-minor-modes
 (abbrev-mode)
 (auto-fill-function)
 (auto-package-update-minor-mode)
 (auto-revert-mode)
 (auto-revert-tail-mode)
 (auto-save-visited-mode)
 (buffer-face-mode)
 (button-mode)
 (cl-old-struct-compat-mode)
 (column-number-mode)
 (comint-fontify-input-mode)
 (compilation-minor-mode)
 (compilation-shell-minor-mode)
 (completion-in-region-mode)
 (context-menu-mode)
 (cursor-face-highlight-mode)
 (cursor-intangible-mode)
 (cursor-sensor-mode)
 (dash-fontify-mode)
 (defining-kbd-macro)
 (diff-auto-refine-mode)
 (diff-minor-mode)
 (dired-hide-details-mode)
 (edit-indirect--overlay)
 (eldoc-mode)
 (electric-layout-mode)
 (electric-quote-mode)
 (elm-format-on-save-mode)
 (elm-indent-mode)
 (elm-indent-simple-mode)
 (flymake-mode)
 (geiser-autodoc-mode)
 (geiser-repl-autoeval-mode)
 (geiser-smart-tab-mode)
 (git-commit-mode)
 (global-auto-revert-mode)
 (global-dash-fontify-mode)
 (global-prettify-symbols-mode)
 (global-semantic-highlight-edits-mode)
 (global-semantic-highlight-func-mode)
 (global-semantic-show-parser-state-mode)
 (global-semantic-show-unmatched-syntax-mode)
 (global-semantic-stickyfunc-mode)
 (global-visual-line-mode)
 (haskell-decl-scan-mode)
 (haskell-indentation-mode)
 (horizontal-scroll-bar-mode)
 (hs-minor-mode)
 (indent-tabs-mode)
 (isearch-mode)
 (jit-lock-debug-mode)
 (lock-file-mode)
 (lost-selection-mode)
 (magit-blame-mode)
 (magit-blame-read-only-mode)
 (magit-blob-mode)
 (magit-wip-after-apply-mode)
 (magit-wip-after-save-local-mode)
 (magit-wip-after-save-mode)
 (magit-wip-before-change-mode)
 (magit-wip-initial-backup-mode)
 (magit-wip-mode)
 (mail-abbrevs-mode)
 (markdown-live-preview-mode)
 (mml-mode)
 (next-error-follow-minor-mode)
 (outline-minor-mode)
 (overwrite-mode)
 (paragraph-indent-minor-mode)
 (paredit-mode)
 (prettify-symbols-mode)
 (racket-hash-lang-repl-mode)
 (racket-smart-open-bracket-mode)
 (racket-xp-mode)
 (read-extended-command-mode)
 (semantic-highlight-edits-mode)
 (semantic-highlight-func-mode)
 (semantic-mode)
 (semantic-show-parser-state-mode)
 (semantic-show-unmatched-syntax-mode)
 (semantic-stickyfunc-mode)
 (server-mode)
 (sh-electric-here-document-mode)
 (shell-command-with-editor-mode)
 (shell-highlight-undef-mode)
 (size-indication-mode)
 (smerge-mode)
 (tab-bar-history-mode)
 (tab-bar-mode)
 (temp-buffer-resize-mode)
 (text-scale-mode)
 (transient-resume-mode)
 (treesit-explore-mode)
 (treesit-inspect-mode)
 (undelete-frame-mode)
 (undo-tree-visualizer-selection-mode)
 (url-handler-mode)
 (use-hard-newlines)
 (vc-dir-git-mode)
 (view-mode)
 (visible-mode)
 (visual-line-mode)
 (which-function-mode)
 (window-divider-mode)
 (with-editor-mode)
 (xref-etags-mode)
 (yas-global-mode)
 (yas-minor-mode))

It occurs to me that Guix may be able to create a reproducible minimal environment that demonstrates the bug. I'll give that a try.

[dpk]

I’ll try this again when I get the chance.

We both use undo-tree-mode and magit-auto-revert-mode. Does disabling one of those help? (I’m clutching at straws, but those are about the only two non-standard minor modes we have in common that I can see.)

[dpk]

Okay, this still doesn’t work with the latest version, and neither undo-tree-mode nor magit-auto-revert-mode seems to be the culprit.

Manually evalling (racket--hash-lang-font-lock-fontify-region (point-min) (point-max)) now also seems to do less highlighting than it did before in my memory (it only highlights strings, keywords, #t, and #f as far as I can tell), but maybe this is unrelated.

I’m trying some other stuff out to see what might be going on here.

[dpk]

Even running Emacs with -Q and loading only package, racket-mode, and racket-hash-lang manually shows the problem.

I wonder if the issue is actually on the Racket end, rather than the Emacs end …

[dpk]

(font-lock-fontify-region (point-min) (point-max)) also does the right thing …

[dpk]

I’ve got it. PR incoming.

[dpk]

(This still doesn’t fix the issue that only strings, keywords, and some constants are being fontified.)

[greghendershott]

I finally realized this might be like Sherlock Holmes and the dog who didn't bark.

Instead of you using some minor-mode that makes this fail?

It looks like I'm using a minor-mode that makes it work. 🤕

[A year ago I'd tested with Emacs 29 built from source and run with -Q, no third-party packages loaded. But I didn't do that again, yesterday, I tried with a my full init file.]

It looks like global-paren-mode from the paren-mode package. When that minor mode is enabled, it does font-lock-flush and font-lock-ensure. Apparently doing that early in the buffer's life, just once, suffices. Possibly it initializes some font-lock state vars behind the scenes (???).

So this is... a lead. I can push a commit to do this in racket-hash-lang-mode's early init. But I want to think this over and make sure I understand why it works. What else might I be overlooking or not understanding.

(For background, racket-hash-lang-mode tries to use jit-lock-mode's ability to fontify portions of the buffer only as necessary. IIUC Emacs' low-level redisplay engine helps do this JIT fontifying. (There's another level where, when our font-lock-some-region function is called, it has to query the back end, and the actual font application happens on an async response to that query.))

[greghendershott]

Oops the page here hadn't refreshed when I left my previous comment. @dpk I see you got all the way to the root of it! 🎉

font-lock-ensure helps only because it calls font-lock-set-defults; using the latter directly as in your PR is better.

[greghendershott]

(This still doesn’t fix the issue that only strings, keywords, and some constants are being fontified.)

That is basically what I would expect. This uses the color-lexer supplied by each lang. That's just a tokenizer. For #lang racket that would be things like '(error comment sexp-comment white-space constant string no-color parenthesis hash-colon-keyword symbol eof other).

The resulting appearance should be like DrRacket -- fairly "plain".

---

If you also enable the minor mode racket-xp-mode, which does a check-syntax kind of pass, it will contribute more colors eventually. Basically for each end of an arrow that DrRacket would draw, there's one of six faces. See the new customization variable racket-xp-binding-font-lock-face-modes.

The default face values give me something roughly like what classic Racket Mode would do -- not exactly the same, but approximately "as colorful" -- for say .rkt and .scrbl files I've dog-fooded so far.

This aspect can be customized, but the out-of-box defaults might not be great for all langs. [As @samth pointed out to me yesterday w.r.t. rhombus and its imports with aliased module names. Not sure if the fix for that is in DrRacket arrows, or on my end, or both. Anyway there may be more issues like that to discover.]

Of course so-called "semantic" highlighting analysis could go further. e.g. I miss classic racket-mode bolding the names in function but not variable definitions; semantically that's arbitrary (functions are just values) but it's handy dammit. 🤷 For now I'm tying to be restrained and lang-agnostic and limit this to using the info from analysis already done by drracket/check-syntax.

Anyway "better semantic highlighting" is probably its own issue, even if the comment thread on this one weren't already so long. 😄 But it's important, too.

[dpk]

(This still doesn’t fix the issue that only strings, keywords, and some constants are being fontified.)

That is basically what I would expect. This uses the color-lexer supplied by each lang. That's just a tokenizer. For #lang racket that would be things like '(error comment sexp-comment white-space constant string no-color parenthesis hash-colon-keyword symbol eof other).

The resulting appearance should be like DrRacket -- fairly "plain".

Weird. I thought I remembered previous versions doing significantly more highlighting (when I manually activated it). But probably I’m misremembering.

[zhaozhixu]

Hi @greghendershott , thanks for your fantastic racket-mode!

I'm using #lang at-exp racket for some code generation tools recently, and I want to customize the indention & syntax highlighting for the specific language (such as C++) snippets inside at-exps to be generated. And I found your recent upgrade of racket-hash-lang-mode could be very useful.

But, I'm using the latest racket-mode-20231222.1554 and still get little highlighting...
The minimal elisp startup code is:

(require 'package)
(package-initialize)
(require 'racket-mode)
(require 'racket-hash-lang)
(add-to-list 'auto-mode-alist '("\\.rkt\\'" . racket-hash-lang-mode))
(add-to-list 'auto-mode-alist '("\\.scrbl\\'" . racket-hash-lang-mode))
(add-to-list 'auto-mode-alist '("\\.rhm\\'" . racket-hash-lang-mode))
(require 'racket-xp)
(add-hook 'racket-hash-lang-mode-hook #'racket-xp-mode)

And I get this with emacs -Q -l startup.el

Update: Tried racket-mode-20231224.2126, still the same.

Package

metadata

(#s(package-desc racket-mode
		 (20231224 2126)
		 "Racket editing, REPL, and more"
		 ((emacs
		   (25 1)))
		 nil nil "/home/zhixu/.emacs.d/elpa/racket-mode-20231224.2126"
		 ((:url . "https://www.racket-mode.com/")
		  (:maintainer "Greg Hendershott")
		  (:maintainers
		   ("Greg Hendershott"))
		  (:authors
		   ("Greg Hendershott" . "racket-mode-author@greghendershott.com"))
		  (:commit . "7b6c9d7f493d301969c6a17a6a4e2171608e119a"))
		 nil)
   #s(package-desc racket-mode
		   (20231222 1554)
		   "Racket editing, REPL, and more"
		   ((emacs
		     (25 1)))
		   nil nil "/home/zhixu/.emacs.d/elpa/racket-mode-20231222.1554"
		   ((:url . "https://www.racket-mode.com/")
		    (:maintainer "Greg Hendershott")
		    (:maintainers
		     ("Greg Hendershott"))
		    (:authors
		     ("Greg Hendershott" . "racket-mode-author@greghendershott.com"))
		    (:commit . "2ad39c6b76e412273719194f22ee77ece3189374"))
		   nil)
   #s(package-desc racket-mode
		   (20230508 1742)
		   "Racket editing, REPL, and more"
		   ((emacs
		     (25 1)))
		   nil nil "/home/zhixu/.emacs.d/elpa/racket-mode-20230508.1742"
		   ((:url . "https://www.racket-mode.com/")
		    (:maintainer "Greg Hendershott")
		    (:maintainers
		     ("Greg Hendershott"))
		    (:authors
		     ("Greg Hendershott" . "racket-mode-author@greghendershott.com"))
		    (:commit . "c6dbe023c688ad1845e1f8ee250f5fa2cf2b8b01"))
		   nil))

package-archives

(("gnu" . "https://elpa.gnu.org/packages/")
 ("nongnu" . "https://elpa.nongnu.org/nongnu/"))

racket--el-source-dir

"/home/zhixu/.emacs.d/elpa/racket-mode-20231224.2126/"

racket--rkt-source-dir

"/home/zhixu/.emacs.d/elpa/racket-mode-20231224.2126/racket/"

System values

emacs-version

"28.2"

major-mode

racket-hash-lang-mode

system-type

gnu/linux

x-gtk-use-system-tooltips

t

display-graphic-p

t

Buffer values

after-change-functions

(jit-lock-after-change t racket--xp-after-change-hook racket--hash-lang-after-change-hook)

before-change-functions

nil

completion-at-point-functions

(racket-xp-complete-at-point)

eldoc-documentation-function

nil

font-lock-defaults

(nil t)

pre-command-hook

(tooltip-hide)

post-command-hook

(global-font-lock-mode-check-buffers global-eldoc-mode-check-buffers mode-local-post-major-mode-change)

post-self-insert-hook

(racket-hash-lang-post-self-insert t)

xref-backend-functions

(racket-xp-xref-backend-function t)

Racket Mode values

racket--cmd-open-p

t

racket-after-run-hook

nil

racket-back-end-configurations

((:directory "/" :racket-program nil :remote-source-dir nil :restart-watch-directories nil :windows nil))

racket-before-run-hook

(racket-ansi-color-context-reset)

racket-browse-url-function

racket-browse-url-using-temporary-file

racket-command-timeout

10

racket-documentation-search-location

"https://docs.racket-lang.org/search/index.html?q=%s"

racket-error-context

medium

racket-hash-lang-mode-hook

(racket-xp-mode)

racket-hash-lang-module-language-hook

nil

racket-hash-lang-token-face-alist

((constant . font-lock-constant-face)
 (error . error)
 (other . font-lock-doc-face)
 (keyword . font-lock-keyword-face)
 (hash-colon-keyword . racket-keyword-argument-face)
 (at . font-lock-doc-face))

racket-history-filter-regexp

"\\`\\s *\\'"

racket-imagemagick-props

nil

racket-images-inline

t

racket-images-keep-last

100

racket-images-system-viewer

"display"

racket-indent-curly-as-sequence

t

racket-indent-sequence-depth

0

racket-logger-config

((cm-accomplice . warning)
 (GC . info)
 (module-prefetch . warning)
 (optimizer . info)
 (racket/contract . error)
 (racket-mode-debugger . info)
 (sequence-specialization . info)
 (* . fatal))

racket-memory-limit

2048

racket-mode-hook

nil

racket-module-forms

"\\s(\\(?:module[*+]?\\|library\\)"

racket-pretty-lambda

nil

racket-pretty-print

t

racket-program

"racket"

racket-repl-buffer-name-function

nil

racket-repl-command-file

"/home/zhixu/.emacs.d/racket-mode/repl.rkt"

racket-repl-history-directory

"~/.emacs.d/racket-mode/"

racket-repl-mode-hook

nil

racket-sexp-comment-fade

0.5

racket-shell-or-terminal-function

racket-shell

racket-show-functions

(racket-show-pseudo-tooltip)

racket-smart-open-bracket-enable

nil

racket-submodules-to-run

((test)
 (main))

racket-use-repl-submit-predicate

nil

racket-xp-add-binding-faces

nil

racket-xp-after-change-refresh-delay

1

racket-xp-highlight-unused-regexp

"^[^_]"

racket-xp-mode-lighter

(:eval
 (racket--xp-mode-lighter))

Minor modes

enabled

((auto-composition-mode)
 (auto-compression-mode)
 (auto-encryption-mode)
 (blink-cursor-mode)
 (file-name-shadow-mode)
 (font-lock-mode)
 (global-eldoc-mode)
 (global-font-lock-mode)
 (indent-tabs-mode)
 (line-number-mode)
 (menu-bar-mode)
 (mouse-wheel-mode)
 (racket-xp-mode)
 (semantic-minor-modes-format)
 (shell-dirtrack-mode)
 (show-paren-mode)
 (tool-bar-mode)
 (tooltip-mode)
 (transient-mark-mode))

Disabled minor modes

disabled

((abbrev-mode)
 (auto-fill-function)
 (auto-fill-mode)
 (auto-save-mode)
 (auto-save-visited-mode)
 (buffer-face-mode)
 (buffer-read-only)
 (button-mode)
 (cl-old-struct-compat-mode)
 (column-number-mode)
 (compilation-minor-mode)
 (compilation-shell-minor-mode)
 (completion-in-region-mode)
 (context-menu-mode)
 (defining-kbd-macro)
 (eldoc-mode)
 (electric-indent-mode)
 (electric-layout-mode)
 (electric-pair-mode)
 (electric-quote-mode)
 (global-prettify-symbols-mode)
 (global-semantic-highlight-edits-mode)
 (global-semantic-highlight-func-mode)
 (global-semantic-show-parser-state-mode)
 (global-semantic-show-unmatched-syntax-mode)
 (global-semantic-stickyfunc-mode)
 (global-visual-line-mode)
 (horizontal-scroll-bar-mode)
 (hs-minor-mode)
 (isearch-mode)
 (jit-lock-debug-mode)
 (lock-file-mode)
 (next-error-follow-minor-mode)
 (overwrite-mode)
 (paragraph-indent-minor-mode)
 (prettify-symbols-mode)
 (racket-hash-lang-repl-mode)
 (racket-smart-open-bracket-mode)
 (semantic-highlight-edits-mode)
 (semantic-highlight-func-mode)
 (semantic-mode)
 (semantic-show-parser-state-mode)
 (semantic-show-unmatched-syntax-mode)
 (semantic-stickyfunc-mode)
 (sh-electric-here-document-mode)
 (size-indication-mode)
 (tab-bar-history-mode)
 (tab-bar-mode)
 (temp-buffer-resize-mode)
 (text-scale-mode)
 (unify-8859-on-decoding-mode)
 (unify-8859-on-encoding-mode)
 (url-handler-mode)
 (use-hard-newlines)
 (view-mode)
 (visible-mode)
 (visual-line-mode)
 (window-divider-mode)
 (xref-etags-mode))

[greghendershott]

@zhaozhixu Hello. You're welcome! :)

Strictly speaking, this issue is about having no font-lock colors at all.

It is normal (unfortunately) for racket-hash-lang-mode not to use very many colors. It colors what the language's lexer reports as different kinds of tokens (strings, comments, etc.). That's it. Comparable to DrRacket colors. Which I do see in your screen shot.

See the end of the doc string for racket-hash-lang-module-hook where it discusses two ways to get more colors:

As another example, if you prefer more colors than just tokens, choices include:

• Enable racket-xp-mode in racket-hash-lang-mode-hook and in the module language hook locally set racket-xp-add-binding-faces:

  (setq-local racket-xp-add-binding-faces t)

OR

• Use some of the regexp search-based fontification from classic racket-mode for rackety module languages:

  (require 'racket-font-lock)
  (if rackety
      (font-lock-add-keywords nil
                              (append racket-font-lock-keywords-2
                                      racket-font-lock-keywords-3))
    (font-lock-remove-keywords nil
                               (append racket-font-lock-keywords-2
                                       racket-font-lock-keywords-3)))

Is either satisfactory for you?

[zhaozhixu]

@zhaozhixu Hello. You're welcome! :)

Strictly speaking, this issue is about having no font-lock colors at all.

It is normal (unfortunately) for racket-hash-lang-mode not to use very many colors. It colors what the language's lexer reports as different kinds of tokens (strings, comments, etc.). That's it. Comparable to DrRacket colors. Which I do see in your screen shot.

See the end of the doc string for racket-hash-lang-module-hook where it discusses two ways to get more colors:

As another example, if you prefer more colors than just tokens, choices include:

• Enable racket-xp-mode in racket-hash-lang-mode-hook and in the module language hook locally set racket-xp-add-binding-faces:

  (setq-local racket-xp-add-binding-faces t)

OR

• Use some of the regexp search-based fontification from classic racket-mode for rackety module languages:

  (require 'racket-font-lock)
  (if rackety
      (font-lock-add-keywords nil
                              (append racket-font-lock-keywords-2
                                      racket-font-lock-keywords-3))
    (font-lock-remove-keywords nil
                               (append racket-font-lock-keywords-2
                                       racket-font-lock-keywords-3)))

Is either satisfactory for you?

Thanks! I enabled racket-xp-mode but forgot to set (setq-local racket-xp-add-binding-faces t). Now I have the colors, but still not as colorful as raw racket-xp-mode. Is that normal?

[greghendershott]

The doc string mentions two options. The first one (which you're trying) draws color based on binding arrows. It's not going to be the same as classic racket-mode; depending on the source code, it could be more or fewer colors.

You can also try the second option listed above. Basically this uses the hard-coded curated list of keywords from racket-mode. This will of course be much closer to racket-mode, for langs like racket.

Maybe you could take another look at the doc string, try these things, and if you're still not satisfied, please open a fresh issue about this?