Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
File watcher! #42
Ok so have a quick look at this!
I have made some quick tweaks to demo an idea.
To demonstrate this, start the website running, and put a break point somewhere in here within
With the site running, edit a file like
I think we can use this to:
Both those features would be big wins for development mode.
Need your input on this in order to work out if its truly viable, and how best to do it.
Also I made some minor tweeks. Smidge often assumes that an input file (
Smidge does still rely on the Physical disk ofcourse, when it is creating the cache / bundle files, and thats absolutely fine.
Just putting a brain dump here, feel free to ignore.
In Dev mode - smidge doesn't do any pre-processing or bundling, it just serves the original files fed to it without any modification.
In which case
So, that aside, in Release mode, Smidge iterates each input file, processes it through the pipleine, and writes the resultant output for the file to a
So far, I have found a way to detect changes to any input file, and re-execute the pipeline against that file, and to update the cached output.
But I have not yet found a way to:
For 2) i think it might be as simple as appending something to the
Thanks for all of this! I've been working through a bunch of this last night looks like I can make this all possible. There's a bit of refactoring needed - but it will be really great as the end result. As you noted before there is no current good way to handle things like TypeScript, etc... because the pre-processor pipeline is ignore in these cases and the original design only catered for normal js/css requests. I'm updating the code now to allow:
With that in place it will give far more granular control over what happens to a bundle and also be able to better handle file watching. I'll keep plugging away at this and let you know how I go.
Very nice! Looking forward to trying it out once its ready. Let me know if I can help out with anything! In NetPack i have written some code to detect whether nodejs is installed, as well as code to detect if npm modules are installed (and optionally install them) that you are welcome to pinch if you need to. Also have a basic typescript compiler there that passes typescript as strings to nodejs rather than file paths.
I've been trying to get some of these changes pushed and have ended up refactoring a few things and noticed that I have a little problem. Currently, each IPreProcessor is has a request scoped lifetime, that's because some of these processors needs access to the current request so that the processor knows how to process things like URLs. This is an issue because the
There's a couple ways to try to fix this:
When performing file watching and wanting to re-process, this happens in a background thread which has no Request. So the only way to get file watching really working is to make the pre-processors that need to know about the request (i.e. some of the css ones) to somehow not rely on it at all.
I've pushed changes to a new branch: https://github.com/Shazwazza/Smidge/tree/file-watching-debug-modes this currently builds but does not run, you can see my commit comments here: 35b9abe
The changes from your PR have been put into this branch but now need to figure out what to do with these request based pre-processors. It's been a long time since I looked at how the css urls stuff works so not sure how possible it will be to run them without knowing the request. It might be possible for them to work just by knowing the site's base URL - but to do that will have to be a custom interface to return the site's base URL and have the value be populated by custom middleware on the first request, and/or configurable by options.
Interesting, I havent got as far as dealing with Css in NetPack yet. It does surprise me that it needs the current request to preprocess the files. I suspect what it really needs is some configuration options like basepath etcs. Allowing a preprocessing step to take some configurable options is the path I have taken in NetPack. I have an "rjs" branch now with a working typescript demo page, and also an experimental rjs preprocessor for optimising Amd modules. I allow each preprocessing pipe to be configured as the pipeline is built so that options that a particular pipe needs like the baseUrl etc and anything else can be passed in ahead of time (i.e before any request). The pipeline is then flushed (processed) on application startup so that all the pre-processing happens before the initial request. A watcher process (if you have watching enabled) then runs as a background task to re-flush pipelines whose inputs changes.
Its funny because i was about to add a Combine Pipe, and then a though occured that as these preprocessed files vare visible via the IFileProvider, I could plug smidge in to handle the bundling aspect of those preprocessed files. Somewhat of a hybrid solution but it might make a fun experiment.
The only problem (in my mind) with smidges bundling capability for me at present is that it doesnt currently handle source maps? I have been looking at a c# .Net solution to that problem and think I have found a solution to handle bundling with sourcemap support purely in C# (no calls out to nodejs). Let me know if yoy are interested more on that topic.
P.s when doing typescript preprocessing, I ended up having to write an NPM package for an in memory typescript compiler that does in memory compilation of a set of typescript files rather than individual teanspile strings individually. I found this was the only way to do a proper compilation of a typescript project that does type checking and other things. Transpiling individual strings loses some safety aspects. Because smidge processes individual files and then combines them at the end, I am not sure how easy doing something like a typescript project compilation (i.e multiple files in a single step) would currently fit into the smidge architecture?
I've got this all working now :) Mostly in this rev: c89986f but also in others.
So file watching is working, it will refresh the disk persisted files too and depending on the cache settings you use you'd see the changes straight away. For example: c89986f#diff-30fbbf9f392f23bc5a5b6d71102e6bd4R99
This change also fixes the problem of debug/production, so now you can have whatever options you want for debug/production including ensuring that composite processing is enabled for debug meaning that the pipeline will execute, so this means things like TypeScript, Less, or whatever will work nicely in debug.
I haven't got around to looking at much more (i.e. source maps) but next up will be further defining the options such as in-memory (non persistent) #46 and also cache busting options, etc...
This will all be part of Smidge 2.0, now just need to find more time ;)
Nice work.. Thats really cool stuff..I'll check it out.