Exploration: supported languages as "singleton values" #52

helixbass · 2023-07-09T18:21:00Z

In this PR:

explore the idea of using fixed-size collections to represent the "all supported languages" collection data structures

Based on more-languages

helixbass · 2023-07-09T18:23:04Z

src/language.rs

-#[derive(Copy, Clone, Debug, Eq, PartialEq, ValueEnum, Hash)]
-pub enum SupportedLanguageName {
+#[derive(Copy, Clone, Debug, Eq, PartialEq, ValueEnum)]
+pub enum SupportedLanguage {


Ok so probably the idea would be that most of the stuff in this file would get generated by some kind of macro "spec"

SupportedLanguage is kind of a "token", just a simple enum that gets used as an index into all of the "all of the supported languages" collections

helixbass · 2023-07-09T18:23:39Z

src/language.rs

-            .unwrap()
-    }
-}
+pub type BySupportedLanguage<T> = [T; 22];


Use fixed-size arrays as the actual data structure for the "all supported languages" collections

Idea for (proc?) macro for generating this stuff:

enumerated_collection! { name => "SupportedLanguage", variants => [ "Rust", ... ] }

would generate:

the SupportedLanguage "token" enum type (I guess with #[derive(clap::ValueEnum)] "baked-in"?)

the BySupportedLanguage type alias

the Index implementation

the From<usize> implementation

the ALL_SUPPORTED_LANGUAGES static collection

a by_supported_language!() proc macro (to be used for "not messing up" the definitions of the languages/names for ignore select collections) that could be invoked like:

pub static ALL_SUPPORTED_LANGUAGES: BySupportedLanguage<SupportedLanguage> = by_supported_language! { Rust => tree_sitter_rust::language(), ... };

(and it would know the right order for those enum variants in the generated array literal regardless of what order you wrote the variants in, and it would fail if you didn't supply exactly the right expected variants)

Actually wait you probably can't generate proc macros from proc macros (at least not that are then able to be "used immediately after") can you?

Ok so maybe by_supported_language!() would be implemented as a declarative macro that just passes along its argument (as a token tree?) + a hard-coded list of the expected enum variants (in the order they should be in the generated array) to an existing proc macro eg by_enumerated_collection!()?

helixbass · 2023-07-09T18:24:08Z

src/language.rs

-    pub name: SupportedLanguageName,
-    pub name_for_ignore_select: &'static str,
+    fn index(&self, index: SupportedLanguage) -> &Self::Output {
+        &self[index as usize]


Basically encapsulate some enum <-> usize massaging

helixbass · 2023-07-09T18:25:33Z

src/language.rs

+            value if value == Self::Json as usize => Self::Json,
+            value if value == Self::Css as usize => Self::Css,
+            value if value == Self::Lua as usize => Self::Lua,
+            _ => unreachable!(),


Obviously this is not technically true (that no usize argument could trigger this code path) but I think we'd be ok panicking if we were trying to convert a non-"known good" usize back into the enum, so using From (instead of eg TryFrom) seems more ergonomically desirable

helixbass · 2023-07-09T18:29:58Z

src/language.rs

+        tree_sitter_query::language(),
+        tree_sitter_json::language(),
+        tree_sitter_css::language(),
+        tree_sitter_lua::language(),


This would obviously be sketchy as far as not messing up the order if the idea wasn't that it would be macro-generated by something where "the relevant things are grouped together"

I guess I decided to make these + the name_for_ignore_select's their own separate data structures (vs in the existing code they are fields on the SupportedLanguage struct) since that seemed like the simplest way to "free up" SupportedLanguage to just be a simple "token" enum

But might be interesting to explore how you'd represent the "main type that's getting passed around" (eg SupportedLanguage) as something with a uniform set of fields too

helixbass · 2023-07-09T18:34:14Z

src/lib.rs

-            .get(&language.name)
-            .unwrap()
-            .get_or_init(|| maybe_get_query(query_source, language.language).map(Arc::new))
+        self.0[language]


So I guess this was one of the goals, getting rid of the "fallible lookup" (ie the .unwrap())

helixbass · 2023-07-09T18:34:53Z

src/lib.rs

                .filter(|(_, value)| value.get().is_some())
+                .map(|(index, value)| -> (SupportedLanguage, _) { (index.into(), value) })


usize <-> enum massaging

Idea: maybe (eg maybe via a newtype wrapper on BySupportedLanguage?) BySupportedLanguage should expose more of a "HashMap-style iteration"?

Since conceptually it is a key-value collection?

Here that would avoid the manual .enumerate() + usize-massaging, which feels like boilerplate

helixbass · 2023-07-09T18:36:26Z

src/project_file_walker.rs

    } else {
-        for language in ALL_SUPPORTED_LANGUAGES.iter() {
-            types_builder.select(language.name_for_ignore_select);
+        for name_for_ignore_select in SUPPORTED_LANGUAGE_NAMES_FOR_IGNORE_SELECT.iter() {


Sort of interesting that with the approach of "moving all of these other fields into their own separate fixed-size data structures" now this usage pattern becomes an iteration over "the data structure that just has that one relevant thing in it"

helixbass · 2023-07-09T18:37:15Z

src/language.rs

-            name: SupportedLanguageName::Lua,
-            name_for_ignore_select: "lua",
-        },
+pub static ALL_SUPPORTED_LANGUAGES: BySupportedLanguage<SupportedLanguage> = {


This one was able to be truly static...

helixbass · 2023-07-09T18:37:47Z

src/language.rs

+    ]
+};
+
+static SUPPORTED_LANGUAGE_LANGUAGES: Lazy<BySupportedLanguage<Language>> = Lazy::new(|| {


...but this one needed a Lazy wrapper since I guess tree_sitter::Language's aren't const?

peterstuart

Nice, this seems like a good approach (assuming that some macros were making it all “safe”).

helixbass · 2023-07-11T14:54:21Z

src/lib.rs

                .filter(|(_, value)| value.get().is_some())
+                .map(|(index, value)| -> (SupportedLanguage, _) { (index.into(), value) })


Idea: maybe (eg maybe via a newtype wrapper on BySupportedLanguage?) BySupportedLanguage should expose more of a "HashMap-style iteration"?

Since conceptually it is a key-value collection?

Here that would avoid the manual .enumerate() + usize-massaging, which feels like boilerplate

helixbass · 2023-07-11T14:55:47Z

src/lib.rs

@@ -222,7 +214,7 @@ pub fn run(args: Args) {
            let query_context = QueryContext::new(
                query,
                capture_index,
-                language.language,
+                get_supported_language_language(language),


Idea: use .into() instead for "type-unique" "mappings" from the token type (eg here SupportedLanguage -> tree_sitter::Language)?

Or actually per other idea, this could just be a (manually-implemented) .language() method on SupportedLanguage?

helixbass · 2023-07-11T15:00:37Z

src/language.rs

 pub static ALL_SUPPORTED_LANGUAGES_BY_NAME_FOR_IGNORE_SELECT: Lazy<
    HashMap<&'static str, SupportedLanguage>,
 > = Lazy::new(|| {
    ALL_SUPPORTED_LANGUAGES
        .iter()
        .map(|supported_language| {
            (
-                supported_language.name_for_ignore_select,
+                get_supported_language_name_for_ignore_select(*supported_language),


Idea: just because with the current approach the "all supported languages collection" for "names for ignore select" is a separate "top-level" data structure doesn't mean that this mapping has to be a top-level function, could just manually add an impl block for SupportedLanguage that exposes an eg name_for_ignore_select() method (whose body does a lookup of self in the "all supported languages collection" for "names for ignore select")?

helixbass · 2023-07-11T15:32:54Z

src/language.rs

-            .unwrap()
-    }
-}
+pub type BySupportedLanguage<T> = [T; 22];


Idea for (proc?) macro for generating this stuff:

enumerated_collection! { name => "SupportedLanguage", variants => [ "Rust", ... ] }

would generate:

the SupportedLanguage "token" enum type (I guess with #[derive(clap::ValueEnum)] "baked-in"?)

the BySupportedLanguage type alias

the Index implementation

the From<usize> implementation

the ALL_SUPPORTED_LANGUAGES static collection

a by_supported_language!() proc macro (to be used for "not messing up" the definitions of the languages/names for ignore select collections) that could be invoked like:

pub static ALL_SUPPORTED_LANGUAGES: BySupportedLanguage<SupportedLanguage> = by_supported_language! { Rust => tree_sitter_rust::language(), ... };

(and it would know the right order for those enum variants in the generated array literal regardless of what order you wrote the variants in, and it would fail if you didn't supply exactly the right expected variants)

Actually wait you probably can't generate proc macros from proc macros (at least not that are then able to be "used immediately after") can you?

Ok so maybe by_supported_language!() would be implemented as a declarative macro that just passes along its argument (as a token tree?) + a hard-coded list of the expected enum variants (in the order they should be in the generated array) to an existing proc macro eg by_enumerated_collection!()?

use fixed-size arrays

50a4cd2

helixbass commented Jul 9, 2023

View reviewed changes

peterstuart approved these changes Jul 9, 2023

View reviewed changes

helixbass commented Jul 11, 2023

View reviewed changes

helixbass mentioned this pull request Jul 11, 2023

Exploration: supported languages as "singleton values": macros #53

Merged

Merge branch 'split-tests-file' into supported-languages-fixed-size

b0401e1

helixbass changed the base branch from more-languages to split-tests-file July 12, 2023 10:41

Base automatically changed from split-tests-file to master July 12, 2023 11:28

Merge branch 'master' into supported-languages-fixed-size

ccda69a

helixbass merged commit 2cf848b into master Jul 13, 2023
2 checks passed

helixbass deleted the supported-languages-fixed-size branch July 13, 2023 21:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exploration: supported languages as "singleton values" #52

Exploration: supported languages as "singleton values" #52

helixbass commented Jul 9, 2023

helixbass Jul 9, 2023

helixbass Jul 9, 2023

helixbass Jul 11, 2023

helixbass Jul 9, 2023

helixbass Jul 9, 2023

helixbass Jul 9, 2023

helixbass Jul 9, 2023

helixbass Jul 9, 2023

helixbass Jul 11, 2023

helixbass Jul 9, 2023

helixbass Jul 9, 2023

helixbass Jul 9, 2023

peterstuart left a comment

helixbass Jul 11, 2023

helixbass Jul 11, 2023

helixbass Jul 11, 2023

helixbass Jul 11, 2023

		.filter(\|(_, value)\| value.get().is_some())
		.map(\|(index, value)\| -> (SupportedLanguage, _) { (index.into(), value) })

Exploration: supported languages as "singleton values" #52

Exploration: supported languages as "singleton values" #52

Conversation

helixbass commented Jul 9, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peterstuart left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment