Introduce TypeList, always export Dependencies as well #221

NyxCode · 2024-02-01T14:50:41Z

No description provided.

NyxCode · 2024-02-01T14:53:05Z

@escritorio-gustavo Here's a proof of concept of the TypeList.
Most notably, I've only added a new method TS::dependency_types() -> impl TypeList here, but I haven't touched the old TS::dependencies() -> Vec<Dependency> yet. I want to run a few tests first, and see if the output of dependency_types() is equivalent to the old dependencies()

ts-rs/src/lib.rs

NyxCode · 2024-02-01T15:01:54Z

Just to illustrate what this would enable:

fn export_dependencies<T: TS>() {
    struct Exporter;
    impl Visitor for Exporter {
        fn visit<T: TS>(&mut self) {
            T::export();
        }
    }

    T::dependency_types().for_each(&mut Exporter);
}

escritorio-gustavo · 2024-02-01T15:05:56Z

fn export_dependencies<T: TS>() {
   struct Exporter;
   impl Visitor for Exporter {
       fn visit<T: TS>(&mut self) {
           T::export();
       }
   }

   T::dependency_types().for_each(&mut Exporter);
}

This looks awesome! Absolutely love it!
I'm heading out right now for the next 2 hours, but when I get back I'll pull this branch to get a better read

NyxCode · 2024-02-01T15:31:06Z

I am very suprised that that's the case, but it seems that dependencies() already gives the same output as dependency_types().
I was sure I fucked something up somewhere, but it seems like I didnt..

I ran that check over most of the test suite, and added a few contrived tests as well, but I cant manage to break it.

NyxCode · 2024-02-01T16:05:43Z

ts-rs/src/lib.rs

+    fn dependencies() -> Vec<Dependency> where Self: 'static {
+        use crate::typelist::TypeVisitor;
+
+        let mut deps: Vec<Dependency> = vec![];
+        struct Visit<'a>(&'a mut Vec<Dependency>);
+        impl<'a> TypeVisitor for Visit<'a> {
+            fn visit<T: TS + 'static + ?Sized>(&mut self) {
+                if let Some(dep) = Dependency::from_ty::<T>() {
+                    self.0.push(dep);
+                }
+            }
+        }
+        Self::dependency_types().for_each(&mut Visit(&mut deps));

-    /// `true` if this is a transparent type, e.g tuples or a list.  
+        deps
+    }


I now completely removed the generation for TS::dependencies() from the macro crate.
I kept the dependencies() function here for now though, with a default implementation (that's not overwritten anywhere), which converts the new TypeList into the old Vec<Dependency>.

NyxCode · 2024-02-01T16:07:30Z

ts-rs/src/typelist.rs

+impl TypeList for () {
+    fn contains<C: Sized>(self) -> bool { false }
+    fn for_each(self, _: &mut impl TypeVisitor) {}
+}


An empty TypeList is just (). That's why TS::dependency_types() -> impl TypeList can have a default implementation with an empty body, since that evaluates to ()

NyxCode · 2024-02-01T16:08:36Z

ts-rs/src/typelist.rs

+impl<T> TypeList for (PhantomData<T>,) where T: TS + 'static + ?Sized {
+    fn contains<C: Sized + 'static>(self) -> bool {
+        TypeId::of::<C>() == TypeId::of::<T>()
+    }
+
+    fn for_each(self, v: &mut impl TypeVisitor) {
+        v.visit::<T>();
+    }
+}


A TypeList with just one element is represented by the tuple (PhantomData<T>,).

NyxCode · 2024-02-01T16:09:56Z

ts-rs/src/typelist.rs

+impl<A, B> TypeList for (A, B) where A: TypeList, B: TypeList {
+    fn contains<C: Sized + 'static>(self) -> bool {
+        self.0.contains::<C>() || self.1.contains::<C>()
+    }
+
+    fn for_each(self, v: &mut impl TypeVisitor) {
+        self.0.for_each(v);
+        self.1.for_each(v);
+    }
+}


Finally, here is where the induction / recursion is happening:
For any two TypeLists A and B, (A, B) is a TypeList as well.

That's honestly the most clever use of the type system I've seen! Especially given how Rust usually doesn't like recursive data structures

😆 I think I've done much more insane things. For example, you can do real arithmetic in in the type system alone

arithmetic in in the type system alone

Wow... just wow

NyxCode · 2024-02-01T17:02:46Z

ts-rs/src/export.rs

+/// Exports `T` to the file specified by the `#[ts(export_to = ..)]` attribute.
+/// Additionally, all dependencies of `T` will be exported as well.
+/// TODO: This might cause a race condition:
+///       If two types `A` and `B` are `#[ts(export)]` and depend on type `C`,
+///       then both tests for exporting `A` and `B` will try to write `C` to `C.ts`.
+///       Since rust, by default, executes tests in paralell, this might cause `C.ts` to be corrupted.
+pub(crate) fn export_type_with_dependencies<T: TS + ?Sized + 'static>() -> Result<(), ExportError> {


I have now added this function, which exports T and recursively all of its dependencies.

The next step here should probably be to actually cause a race condition, and then figure out how to fix it (maybe by putting a static MUTEX: Mutex<()> in TS? Or run the tests with RUST_TEST_THREADS=1 to disable paralell execution?)

Oh yeah - that fixed both E2E tests ^^

Oh yeah that's gonna be a hard one to handle! Especially since the compiler, which usually protects us from race conditions, can't do much for us here in test land

Interestingly enough, I cant get it to fail, even with 16 types.
I think one easy solution might be to first write to a temp file, and then rename it. Gotta look into that, but I think renaming is atomic on all platforms.

Oh yeah - that fixed both E2E tests ^^

What fixed it? The mutex or the RUST_TEST_THREADS?

Nothing! I just tried to provoke the error, by exporting lots of types and setting RUST_TEST_THREADS=16, but nothing broke. Maybe that's because the files are relatively small, and get written in just one syscall. I'll try exporting larger types, maybe that'll break it.

Yeah, idk. I made the types now 17MB large with 2 million numbers in them. cargo test still refuses to corrupt them..

escritorio-gustavo · 2024-02-01T17:04:35Z

I am very suprised that that's the case, but it seems that dependencies() already gives the same output as dependency_types().
I was sure I fucked something up somewhere, but it seems like I didnt..

I ran that check over most of the test suite, and added a few contrived tests as well, but I cant manage to break it.

I too refuse to believe my code is ever working first time xD

ts-rs/src/export.rs

escritorio-gustavo · 2024-02-01T17:55:58Z

It didn't fail CI! Is this somehow just thread safe? lol

NyxCode · 2024-02-01T18:02:44Z

It didn't fail CI! Is this somehow just thread safe? lol

Really curious about this as well. My theory is the following

Thread 1 creates the file (File::create) and starts writing to it
Thread 2 creates the file (File::create), but to the OS, it's a completely different inode.
Thread 1 continues to write to the old file handle, not disturbing thread 2

It's hard to find a definite resource on this.

escritorio-gustavo · 2024-02-01T18:07:21Z

It's hard to find a definite resource on this.

Yeah, I'm trying to read on std::fs::write but I can't find anything in a similar situation, where the threads aren't coordinated by a single centralized place

NyxCode · 2024-02-01T18:42:47Z

An easy fix for the concurrent write issue would be to change export_type_to:

pub(crate) fn export_type_to<T: TS + ?Sized + 'static, P: AsRef<Path>>(
    path: P,
) -> Result<(), ExportError> {
    static FILE_LOCK: Mutex<()> = Mutex::new(());
    
    // ...

    let lock = FILE_LOCK.lock().unwrap();
    std::fs::write(path.as_ref(), buffer)?;
    drop(lock);

    Ok(())
}

That lock would be shared by every file export, so no two files would be written concurrently ever.

NyxCode · 2024-02-01T18:44:24Z

Though that issue is purely theoretical right now, so we could just do it & see if any1 encounters an issue - idk.

escritorio-gustavo · 2024-02-01T18:46:55Z

An easy fix for the concurrent write issue would be to change export_type_to:

pub(crate) fn export_type_to<T: TS + ?Sized + 'static, P: AsRef<Path>>(
    path: P,
) -> Result<(), ExportError> {
    static FILE_LOCK: Mutex<()> = Mutex::new(());
    
    // ...

    let lock = FILE_LOCK.lock().unwrap();
    std::fs::write(path.as_ref(), buffer)?;
    drop(lock);

    Ok(())
}

That lock would be shared by every file export, so no two files would be written concurrently ever.

Wait, this works? I always thought the mutex had to be shared between threads (hence the whole Arc<Mutex<T>> thing people like to talk about).

escritorio-gustavo · 2024-02-01T18:48:21Z

Or am I just missing something?
As far as I can tell every thread is gonna run export_type_to on its own right? So each thread will have its own Mutex, wihch won't be shared

NyxCode · 2024-02-01T18:53:52Z

Or am I just missing something? As far as I can tell every thread is gonna run export_type_to on its own right? So each thread will have its own Mutex, wihch won't be shared

No, I don't think that's right. Because it's static, there is only one mutex somewhere in memory.
Because that is the case, you could do

static SOMETHING: u32 = 125;
let reference: &'static u32 = &SOMETHING;

The reference can be static because the static lives forever.

Now, I had to look that up, but that the function export_type_to is generic doesn't change that:

A static item defined in a generic scope (for example in a blanket or default implementation) will result in exactly one static item being defined, as if the static definition was pulled out of the current scope into the module. There will not be one item per monomorphization.

NyxCode · 2024-02-01T18:56:42Z

You often see Arc<Mutex> because the Mutex does not live forever normally, when not in a static. So the Arc is there to reference-count and clean the mutex when it's no longer used.
Besides using static, an other way to get rid of the Arc is by leaking it:

let x: &'static Mutex<()> = Box::leak(Box::new(Mutex::new(())));

Then, just like the static, you've got a &'static reference to it, which you can pass to every thread you want.

escritorio-gustavo · 2024-02-01T19:16:11Z

No, I don't think that's right. Because it's static, there is only one mutex somewhere in memory.

Oh now I get it! And this is definetly correct, I set up a small test to check with a function that sleeps and 20 tests calling it with and without the Mutex and also switching static with let

With static the tests really do wait for each other

escritorio-gustavo · 2024-02-01T19:17:47Z

In this test I also created a file with 20 paragraphs of lorem ipsum. Even without the Mutex I couldn't cause it to corrupt the file either

escritorio-gustavo · 2024-02-01T19:25:08Z

Okay, I've got the following:

#[cfg(test)]
fn do_it() {
    std::fs::write(PATH, CONTENT).unwrap();
}
#[test]
fn write_1() {
    for i in 0..10 {std::thread::spawn(|| do_it());}
}

#[test]
fn write_2() {
    for i in 0..10 {std::thread::spawn(|| do_it());}
}

#[test]
fn write_3() {
    for i in 0..10 {std::thread::spawn(|| do_it());}
}

// ...

#[test]
fn write_20() {
    for i in 0..10 {std::thread::spawn(|| do_it());}
}

const PATH: &str = "./foo.txt";
const CONTENT: &str = "VERY LONG STRING";

escritorio-gustavo · 2024-02-01T19:26:45Z

This is producing an empty file

NyxCode · 2024-02-01T19:29:36Z

Alright, awesome! That makes me feel less bad putting a Mutex in there!
Though I think it's remarkable that it took 200 threads to get it to break.

escritorio-gustavo · 2024-02-01T19:33:10Z

Hold up, the way I did it there breaks, but when I join the threads, it's fine

#[test]
fn write_1() {
    let mut v = vec![];
    for i in 0..10 {
        v.push(std::thread::spawn(|| do_it()));
    }
    for i in v {
        i.join();
    }
}

escritorio-gustavo · 2024-02-01T19:42:11Z

Sitll even if std::fs::write happens to be thread safe when writing the same data to the same file, that's not explicitly documented to be the case and we probably shouldn't rely on it

escritorio-gustavo · 2024-02-01T19:43:17Z

e2e/workspace/crate1/src/lib.rs

@@ -2,5 +2,5 @@ use ts_rs::TS;

 #[derive(TS)]
 pub struct Crate1 {
-    pub x: i32
+    pub x: [[[i32; 128]; 128]; 128],


I like how the fact that this generates stupidly large TS types ended up being useful lol

NyxCode · 2024-02-01T19:55:46Z

From my side this is ready for merge!
I'd like to aditionally expand the E2E tests a bit, but i'll do that in an other PR.
So maybe you could take a last look, and if everything's alright merge this.

Again, huge thanks for exploring the problem space with me.
It's actually fun maintaining OS stuff if there's some1 to do it with ❤️

escritorio-gustavo · 2024-02-01T19:57:51Z

It really is! I love to have someone to about this kinda stuff with!

escritorio-gustavo · 2024-02-01T19:59:15Z

I think it's ready for merging as well, though we have to remember to also merge #218 after this, as this is going to the e2e branch

NyxCode · 2024-02-01T20:01:28Z

Alright! A bit akward, but whatever ^^

NyxCode · 2024-02-03T19:05:10Z

Something which I completely forgot to pay attention to are self-referential structs!
I've ran some tests (7.1.1 vs master), and this is how the behavious changed:

`this: Box<Self>`	on 7.1.1	on master
as-is	✅	✅
`#[ts(flatten)]`	❌ (runtime) stack overflow	❌ (compiler) "overflow evaluating the requirement"
`#[ts(inline)]`	❌ (runtime) stack overflow	❌ (compiler) "overflow evaluating the requirement"
`#[ts(type = "any", flatten)]`	❌ (macro) "... is not compatible ..."	❌ (macro) "... is not compatible ..."
`#[ts(type = "any", inline)]`	⚠️ works, but shouldn't	⚠️ works, but shouldn't

So what failed before with a stack overflow now fails at compile time - That's pretty cool, i think!

error[E0275]: overflow evaluating the requirement `<T as TS>::{opaque#0} == _`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0275`.
error: could not compile `ts-rs` (test "self_referential") due to 2 previous errors

Also, I realized that both 7.1.1 and master accept #[ts(type = "..", inline)] - As far as I can tell, inline never does anything when used together with type, right? So maybe we should emit an error for this, like we do for #[ts(type = "..", flatten)]

NyxCode · 2024-02-03T20:44:22Z

Found something interesting:

#[derive(TS)]
struct X<T> {
    a: T,
    b: Vec<X<(i32, T)>>
}

fails to compile with

thread 'rustc' has overflowed its stack
error: could not compile `ts-rs` (test "generics")

Tested both on 7.1.1 and master, so this is not a regression we introduces (recently). Might be interesting to find out why it's happening though. In a perfect world, it should compile and product

type X<T> = {
  a: T,
  b: Array<X<[number, T]>>,
}

Will open an issue to keep track of this.

escritorio-gustavo · 2024-02-05T11:08:02Z

So what failed before with a stack overflow now fails at compile time - That's pretty cool, i think!

Agreed, if something is not gonna work it's far better to know about it at compile time

Also, I realized that both 7.1.1 and master accept #[ts(type = "..", inline)] - As far as I can tell, inline never does anything when used together with type, right? So maybe we should emit an error for this, like we do for #[ts(type = "..", flatten)]

Yeah, #[ts(type = "...", inline)] doesn't really make sense, we should emit an error for it

NyxCode · 2024-04-11T20:44:36Z

ts-rs/src/export.rs

+    // Lock to make sure only one file will be written at a time.
+    // In the future, it might make sense to replace this with something more clever to only prevent
+    // two threads from writing the **same** file concurrently.
+    static FILE_LOCK: Mutex<()> = Mutex::new(());
+


@escritorio-gustavo Regarding the lock we introduced here:

I've experimented with replacing this with a static EXPORTED: Mutex<Option<HashSet<PathBuf>>> to track which files were written, and to skip an export if the file was already written to.

The case in our test suite where this is most helpfull is generics.rs, where 30 distinct types are being exported while 37 unnecesarry exports are being skipped.

That being said, I don't think we actually need to do that. generics.rs ran in 50ms before, and runs in 32ms after this (though with much higher variance). That seems plenty fast.

It definitely seems fast enough as is, but if we can reduce the amount of fs writes without any negative impact, I think it's a good idea

Maybe push that into #295 as a part of the test performance improvements? That PR is more focused on improving build time, but improving runtime is also great!

implement POC

3fa1d4b

NyxCode changed the base branch from main to e2e February 1, 2024 14:51

NyxCode commented Feb 1, 2024

View reviewed changes

ts-rs/src/lib.rs Outdated Show resolved Hide resolved

fix missing import

3e2b8cf

remove old dependency implementation, fix warnings

2375fbf

NyxCode commented Feb 1, 2024

View reviewed changes

recursively export all dependencies

ab569e4

NyxCode commented Feb 1, 2024

View reviewed changes

escritorio-gustavo reviewed Feb 1, 2024

View reviewed changes

ts-rs/src/export.rs Outdated Show resolved Hide resolved

use stupidly large types in E2E test - maybe it fails on CI?

646ccd9

escritorio-gustavo added 2 commits February 1, 2024 15:07

Separate recursive exporting into mod

3de39ab

Merge branch 'typelist' of github.com:Aleph-Alpha/ts-rs into typelist

d9464e3

escritorio-gustavo reviewed Feb 1, 2024

View reviewed changes

introduce mutex for file exports, change array tuple limit to 64

dcec4d8

NyxCode changed the title ~~[Proof of Concept] Typelist~~ Introduce TypeList, always export Dependencies as well Feb 1, 2024

NyxCode marked this pull request as ready for review February 1, 2024 19:51

NyxCode linked an issue Feb 1, 2024 that may be closed by this pull request

generating invalid TypeScript import path with using Cargo workspace #111

Closed

escritorio-gustavo merged commit 8eb28e5 into e2e Feb 1, 2024
8 checks passed

escritorio-gustavo deleted the typelist branch February 1, 2024 20:01

escritorio-gustavo mentioned this pull request Feb 2, 2024

generating invalid TypeScript import path with using Cargo workspace #111

Closed

escritorio-gustavo mentioned this pull request Feb 7, 2024

Feat: add default export directory override #195

Merged

1 task

NyxCode mentioned this pull request Feb 29, 2024

Bug: Type Parameters are not imported #241

Merged

NyxCode commented Apr 11, 2024

View reviewed changes

escritorio-gustavo mentioned this pull request Apr 12, 2024

CLI initial concept #304

Open

12 tasks

Introduce TypeList, always export Dependencies as well #221

Introduce TypeList, always export Dependencies as well #221

Conversation

NyxCode commented Feb 1, 2024

NyxCode commented Feb 1, 2024

NyxCode commented Feb 1, 2024

escritorio-gustavo commented Feb 1, 2024

NyxCode commented Feb 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

escritorio-gustavo commented Feb 1, 2024

escritorio-gustavo commented Feb 1, 2024

NyxCode commented Feb 1, 2024

escritorio-gustavo commented Feb 1, 2024

NyxCode commented Feb 1, 2024

NyxCode commented Feb 1, 2024

escritorio-gustavo commented Feb 1, 2024

escritorio-gustavo commented Feb 1, 2024

NyxCode commented Feb 1, 2024

NyxCode commented Feb 1, 2024

escritorio-gustavo commented Feb 1, 2024

escritorio-gustavo commented Feb 1, 2024

escritorio-gustavo commented Feb 1, 2024 • edited Loading

escritorio-gustavo commented Feb 1, 2024

NyxCode commented Feb 1, 2024

escritorio-gustavo commented Feb 1, 2024

escritorio-gustavo commented Feb 1, 2024

escritorio-gustavo Feb 1, 2024 • edited Loading

Choose a reason for hiding this comment

NyxCode commented Feb 1, 2024

escritorio-gustavo commented Feb 1, 2024

escritorio-gustavo commented Feb 1, 2024 • edited Loading

NyxCode commented Feb 1, 2024

NyxCode commented Feb 3, 2024

NyxCode commented Feb 3, 2024

escritorio-gustavo commented Feb 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

escritorio-gustavo Apr 12, 2024 • edited Loading

Choose a reason for hiding this comment

NyxCode commented Feb 1, 2024 •

edited

Loading

escritorio-gustavo commented Feb 1, 2024 •

edited

Loading

escritorio-gustavo Feb 1, 2024 •

edited

Loading

escritorio-gustavo commented Feb 1, 2024 •

edited

Loading

escritorio-gustavo Apr 12, 2024 •

edited

Loading