Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] LuaSkin and coroutines #2308

Closed
wants to merge 80 commits into from

Conversation

asmagill
Copy link
Member

Putting it here for anyone interested to review.

This has #2307 already applied and will continue to track it if I find additional dispatch related or other bugs during this conversion.

It's minimally tested, but doesn't break anything if you're not using coroutines. So far, only some of the modules have been converted to the new LuaSkin constructor -- check the commits as I'm doing each module as a separate commit.

Coroutines do work now, as long as you stick to pure lua or functions/methods that only come from or depend on the converted modules... any unconverted module, even if it's loaded as a dependency is likely to still cause a crash if used from within a coroutine.

This is a work in progress and is not yet ready for formal testing or use.

@latenitefilms
Copy link
Contributor

Amazing! Will test on Monday.

@latenitefilms
Copy link
Contributor

@asmagill - Hope you're keeping safe and healthy in these crazy times!

Just checking in... what are the chances of merging this code into the Hammerspoon master branch? @stickler-ci issues aside, it seems like this pull request basically just fixes coroutine support without breaking any existing Hammerspoon features - so it could be safely merged without causing any existing users any issues?

The Film & Television in Australia is basically now completely on hold for the foreseeable future, so apologies in advance, as I'll no doubt have a lot more time on my hands to poke at Hammerspoon & CommandPost!

@asmagill
Copy link
Member Author

I consider it such and have been using it on both Mojave and Catalina since this was originally put up here.

As long as you do not use coroutines, this doesn't introduces any new bugs (that wouldn't also be introduced by #2307) and does in fact fix a few bugs (as does #2307).

If you do use coroutines... well, it is possible that I've missed one or two places where I should be using withState:NULL instead of withState:L (the opposite is unlikely because it's obvious where L isn't defined in callbacks, delegate methods, etc. What is less obvious is when a function queues an action with dispatch_async -- since L is defined outside of the block, no compile time error occurs). But this was one of the things I specifically looked for, so hopefully I haven't missed any.

The real test would be for people to use it and code with coroutines. I've started a few, but most of the ones I want to do to replace things that are a little slow or laggy also require hs._asm.axuielement, so I've been cleaning that up a bit as well. (I have ideas on possibly rewriting hs.window and portions of hs.webview at some point to use coroutines, but those definitely require hs._asm.axuielement to be in core first)

@cmsj, the real question is what level of example or testing will make you comfortable incorporating this? I can put together a short write up (I meant to already but got distracted) so that it's obvious to those of us working directly on the Objective-C extensions what changes should be made and where. Beyond that... what would help your comfort level?

I don't think many of our users will be rushing to use coroutines, so most won't notice a difference... it's not an obvious way of thinking about things until you run into slow or laggy code, and most of Hammerspoon is designed around a trigger-response model to do something short and specific in response to a pre-defined occurrence (a key stroke, a timer, an event, etc.). But for those of us trying to make Hammerspoon seem more responsive or who do write more complex, longer running code (@latenitefilms and his crew), it becomes a very useful addition to support them -- much simpler (and certainly much more understandable to others who may want to modify it) in many cases than rewriting a threaded version solely in Objective-C.

As a reminder, at present, if anyone does use coroutines that either use any Hammerspoon specific Objective-C addition or that resume/yield in different code "blocks" (i.e. there is any Hammerspoon application idle time between them), then Hammerspoon currently crashes (and often in a way that the exception handler can't catch) and we have yet to discover any way to detect or prevent this. Ultimately, we need this pull, or we need to remove the coroutine library and make it clear that we're only supporting a subset of Lua.

@latenitefilms
Copy link
Contributor

Sweet - will await @cmsj's verdict! Thanks heaps @asmagill!

I look VERY forward to when hs._asm.axuielement eventually finds its way into Hammerspoon's core too!

Thanks again for all your amazing hard work!

@latenitefilms
Copy link
Contributor

@asmagill - I just finally worked out how to merge this branch into CommandPost, so will keep you posted on how we go with it! We're also now using hs._asm.axuielement 0.7.3-coroutineFriendly.

@latenitefilms
Copy link
Contributor

You already know this, but just FYI - we're also using these extensions:

2020-03-22 09:18:28: 09:18:28 ** Warning:   LuaSkin: Deprecated LuaSkin method [LuaSkin shared] invoked by `/Users/chrishocking/Github/CommandPost/src/extensions/hs/_asm/cfpreferences/internal.so`. Please notify developer of module to upgrade as this method is unsafe for use with coroutines and may disappear in a future Hammerspoon release.
2020-03-22 09:18:28: 09:18:28 ** Warning:   LuaSkin: Deprecated LuaSkin method [LuaSkin shared] invoked by `/Users/chrishocking/Github/CommandPost/src/extensions/hs/_asm/xml/internal.so`. Please notify developer of module to upgrade as this method is unsafe for use with coroutines and may disappear in a future Hammerspoon release.

@latenitefilms
Copy link
Contributor

Your getMenuItems implementation also works way better/faster and doesn't seem to crash - woohoo!

@latenitefilms
Copy link
Contributor

Just spotted this too:

2020-03-22 10:40:51: 10:40:51 ** Warning:   LuaSkin: Deprecated LuaSkin method [LuaSkin shared] invoked by `/Users/chrishocking/Github/CommandPost-App/build/CommandPost.app/Contents/Resources/extensions/hs/image/internal.so`. Please notify developer of module to upgrade as this method is unsafe for use with coroutines and may disappear in a future Hammerspoon release.

@asmagill
Copy link
Member Author

At present, the warnings are just that -- warnings, and they will only appear once per application launch. As long as you don't use methods from those specific modules (or the specific function/method, if you can identify which it is within image that got overlooked) within a coroutine, it won't be a problem in the short run.

The cfpreferences and xml ones I'll try and get to tomorrow and put up coroutine safe versions of those.

The image one I suspect may be an artifact of your merge if you had to do any hand tweaking... I'm not seeing a place where [LuaSkin shared] is invoked in the image module within my pull. I'm assuming if I look at the CommandPost github page I can find your version with this pull applied (either in the main branch or a separate one)? I can look at that tomorrow as well.

I'm hoping to have a writeup added to the wiki sometime Sunday evening/Monday morning as well -- I've finished (though not pushed) a document outlining the problem and giving a brief introduction of coroutines and why I think we haven't really seen an issue before, but I'm only about half way done with the companion page which describes how to properly fix existing modules and write them going forwards based upon the pull I am recommending.

@latenitefilms
Copy link
Contributor

Ah, yes, you're absolutely right - I had hs.image:getLoupedeckArray() added to our branch. Will correct! Thank you!

@asmagill
Copy link
Member Author

@cmsj, @latenitefilms -- it's a first draftish kind of thing, but I've documented the problem and how to apply this proposed solution going forward. Note that all of the fixes outlined in the second document have already been applied in this pull to the core modules; the documentation is for future modules and for converting the third party ones that still exist outside of core at the moment.

@latenitefilms
Copy link
Contributor

This is AMAZING. THANK YOU!!

--- * unlike `coroutine.yield`, this function does not allow the passing of (new) information to or from the coroutine while it is running; this function is to allow long running tasks to yield time to the Hammerspoon application so other timers and scheduled events can occur without requiring the programmer to add code for an explicit resume.
---
--- * this function is added to the lua `coroutine` library as `coroutine.applicationYield` as an alternative name.
local resumeTimers = {}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(W241) variable 'resumeTimers' is mutated but never accessed

--- * this function is added to the lua `coroutine` library as `coroutine.applicationYield` as an alternative name.
local resumeTimers = {}

hs.coroutineApplicationYield = function(delay)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(W122) setting read-only field 'coroutineApplicationYield' of global 'hs'

end
end

coroutine.applicationYield = hs.coroutineApplicationYield

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(W142) setting undefined field 'applicationYield' of global 'coroutine'


return setmetatable(module, {
-- only invoked if key not already in `module`
__index = function(self, key)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(W212) unused argument 'self'

@latenitefilms
Copy link
Contributor

The cfpreferences and xml ones I'll try and get to tomorrow and put up coroutine safe versions of those.

Just spotted them - thanks heaps @asmagill !!! Will let you know if we run into any issues.

@cmsj
Copy link
Member

cmsj commented Apr 7, 2020

For now this is merged by hand, along with #1646, but a lot of tests are failing, so I must have broken something in the merge. I'll investigate before I revert.

@cmsj
Copy link
Member

cmsj commented Apr 8, 2020

Most of the test failures were LuaSkin, which I think I've resolved in 54a2b09 but the lifecycle stuff around this is very tricky and subtle, so it's possible I've made things worse in other ways.

@asmagill
Copy link
Member Author

asmagill commented Apr 8, 2020

FWIW I never got it to run the test suite correctly myself, nor when Travis did it; but when I ran the failing tests individually in XCode, they all passed... I don't know the testing environment that well, and you'd mentioned that Travis sometimes fails in weird ways as well, so I haven't really done much on that front since... if you figure out something I missed or should be doing, don't hesitate to let me know.

@latenitefilms
Copy link
Contributor

@cmsj - Awesome! Now that this is all merged into the master Hammerspoon branch, I'll attempt to manually merged across into CommandPost-App and see what breaks.

@cmsj
Copy link
Member

cmsj commented Apr 8, 2020

Ok, and now we're back to the same 4 test failures that I was getting before, so I'm going to close this PR out.

@asmagill I think I got all of the [LuaSkin shared] usages in hs.appliation/uielement/window and the new shared HSuicore.m, sorted out in my rewrite.

@cmsj cmsj closed this Apr 8, 2020
@latenitefilms
Copy link
Contributor

@cmsj & @asmagill - I've attempted to merge these changes into CommandPost-App, but I'm getting errors when I try and build, so I've screwed something up:

** BUILD FAILED **


The following build commands failed:
	CompileC /Users/chrishocking/Library/Developer/Xcode/DerivedData/Hammerspoon-eybhxyeverxtpibujwghieyonrun/Build/Intermediates.noindex/Hammerspoon.build/Release/Hammerspoon.build/Objects-normal/x86_64/MJLua.o /Users/chrishocking/Github/CommandPost-App/Hammerspoon/MJLua.m normal x86_64 objective-c com.apple.compilers.llvm.clang.1_0.compiler
(1 failure)
make: *** [build/CommandPost.app] Error 65

Last line in Release-build.log is:

Ld /Users/chrishocking/Library/Developer/Xcode/DerivedData/Hammerspoon-eybhxyeverxtpibujwghieyonrun/Build/Products/Release/libapplication.dylib normal x86_64 (in target 'application' from project 'Hammerspoon')
    cd /Users/chrishocking/Github/CommandPost-App
    /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -target x86_64-apple-macos10.12 -dynamiclib -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -L/Users/chrishocking/Library/Developer/Xcode/DerivedData/Hammerspoon-eybhxyeverxtpibujwghieyonrun/Build/Products/Release -F/Users/chrishocking/Library/Developer/Xcode/DerivedData/Hammerspoon-eybhxyeverxtpibujwghieyonrun/Build/Products/Release -F. -F.fauxpas -FHammerspoon -FHammerspoon\ Tests -FHammerspoon.xcworkspace -FHammerspoonUITests -FLuaSkin -FPods -Fbuild -Fextensions -Fscripts -FHammerspoon.xcworkspace/xcshareddata -FHammerspoon.xcworkspace/xcuserdata -FLuaSkin/LuaSkin -FLuaSkin/LuaSkinTests -FLuaSkin/lua-5.3.5 -FPods/ASCIImage -FPods/CocoaAsyncSocket -FPods/CocoaHTTPServer -FPods/CocoaLumberjack -FPods/Crashlytics -FPods/Fabric -FPods/Headers -FPods/Local\ Podspecs -FPods/MIKMIDI -FPods/PocketSocket -FPods/SocketRocket -FPods/Sparkle -FPods/Target\ Support\ Files -Fextensions/_coresetup -Fextensions/alert -Fextensions/appfinder -Fextensions/applescript -Fextensions/application -Fextensions/audiodevice -Fextensions/audiounit -Fextensions/base64 -Fextensions/battery -Fextensions/bonjour -Fextensions/brightness -Fextensions/caffeinate -Fextensions/canvas -Fextensions/chooser -Fextensions/console -Fextensions/crash -Fextensions/deezer -Fextensions/dialog -Fextensions/distributednotifications -Fextensions/doc -Fextensions/dockicon -Fextensions/drawing -Fextensions/eventtap -Fextensions/expose -Fextensions/fnutils -Fextensions/fs -Fextensions/geometry -Fextensions/grid -Fextensions/hash -Fextensions/hid -Fextensions/hints -Fextensions/host -Fextensions/hotkey -Fextensions/http -Fextensions/httpserver -Fextensions/image -Fextensions/inspect -Fextensions/ipc -Fextensions/itunes -Fextensions/javascript -Fextensions/json -Fextensions/keycodes -Fextensions/layout -Fextensions/location -Fextensions/logger -Fextensions/math -Fextensions/menubar -Fextensions/messages -Fextensions/midi -Fextensions/milight -Fextensions/mjomatic -Fextensions/mouse -Fextensions/network -Fextensions/noises -Fextensions/notify -Fextensions/osascript -Fextensions/pasteboard -Fextensions/pathwatcher -Fextensions/plist -Fextensions/redshift -Fextensions/screen -Fextensions/settings -Fextensions/sharing -Fextensions/socket -Fextensions/sound -Fextensions/spaces -Fextensions/speech -Fextensions/spoons -Fextensions/spotify -Fextensions/spotlight -Fextensions/sqlite3 -Fextensions/streamdeck -Fextensions/styledtext -Fextensions/tabs -Fextensions/tangent -Fextensions/task -Fextensions/timer -Fextensions/uielement -Fextensions/urlevent -Fextensions/usb -Fextensions/utf8 -Fextensions/vox -Fextensions/watchable -Fextensions/websocket -Fextensions/webview -Fextensions/wifi -Fextensions/window -Fscripts/docs -FHammerspoon.xcworkspace/xcshareddata/xcdebugger -FHammerspoon.xcworkspace/xcshareddata/xcschemes -FHammerspoon.xcworkspace/xcuserdata/chrishocking.xcuserdatad -FLuaSkin/lua-5.3.5/doc -FLuaSkin/lua-5.3.5/src -FPods/ASCIImage/Core -FPods/CocoaAsyncSocket/Source -FPods/CocoaHTTPServer/Core -FPods/CocoaHTTPServer/Extensions -FPods/CocoaLumberjack/Classes -FPods/Crashlytics/OSX -FPods/Fabric/OSX -FPods/Headers/Private -FPods/Headers/Public -FPods/MIKMIDI/Source -FPods/PocketSocket/PocketSocket -FPods/SocketRocket/SocketRocket -FPods/Sparkle/bin -FPods/Target\ Support\ Files/ASCIImage -FPods/Target\ Support\ Files/CocoaAsyncSocket -FPods/Target\ Support\ Files/CocoaHTTPServer -FPods/Target\ Support\ Files/CocoaLumberjack -FPods/Target\ Support\ Files/Crashlytics -FPods/Target\ Support\ Files/Fabric -FPods/Target\ Support\ Files/MIKMIDI -FPods/Target\ Support\ Files/PocketSocket -FPods/Target\ Support\ Files/Pods-Hammerspoon -FPods/Target\ Support\ Files/SocketRocket -FPods/Target\ Support\ Files/Sparkle -Fextensions/doc/hsdocs -Fextensions/drawing/color -Fextensions/host/locale -Fextensions/ipc/cli -Fextensions/mouse/manymouse -Fextensions/network/ping -Fextensions/spoons/templates -Fscripts/docs/bin -Fscripts/docs/templates -FPods/CocoaAsyncSocket/Source/GCD -FPods/CocoaHTTPServer/Core/Categories -FPods/CocoaHTTPServer/Core/Mime -FPods/CocoaHTTPServer/Core/Responses -FPods/CocoaHTTPServer/Extensions/WebDAV -FPods/CocoaLumberjack/Classes/CLI -FPods/CocoaLumberjack/Classes/Extensions -FPods/Headers/Private/ASCIImage -FPods/Headers/Private/CocoaAsyncSocket -FPods/Headers/Private/CocoaHTTPServer -FPods/Headers/Private/CocoaLumberjack -FPods/Headers/Private/MIKMIDI -FPods/Headers/Private/PocketSocket -FPods/Headers/Private/SocketRocket -FPods/Headers/Public/ASCIImage -FPods/Headers/Public/CocoaAsyncSocket -FPods/Headers/Public/CocoaHTTPServer -FPods/Headers/Public/CocoaLumberjack -FPods/Headers/Public/MIKMIDI -FPods/Headers/Public/PocketSocket -FPods/Headers/Public/SocketRocket -FPods/Sparkle/bin/old_dsa_scripts -Fextensions/doc/hsdocs/hsminweb -Fscripts/docs/templates/Hammerspoon.docset -Fscripts/docs/templates/Hammerspoon.docset/Contents -Fscripts/docs/templates/Hammerspoon.docset/Contents/Resources -Fscripts/docs/templates/Hammerspoon.docset/Contents/Resources/Documents -filelist /Users/chrishocking/Library/Developer/Xcode/DerivedData/Hammerspoon-eybhxyeverxtpibujwghieyonrun/Build/Intermediates.noindex/Hammerspoon.build/Release/application.build/Objects-normal/x86_64/application.LinkFileList -install_name /usr/local/lib/libapplication.dylib -Xlinker -object_path_lto -Xlinker /Users/chrishocking/Library/Developer/Xcode/DerivedData/Hammerspoon-eybhxyeverxtpibujwghieyonrun/Build/Intermediates.noindex/Hammerspoon.build/Release/application.build/Objects-normal/x86_64/application_lto.o -fobjc-arc -fobjc-link-runtime -undefined dynamic_lookup -framework LuaSkin -compatibility_version 1 -current_version 1 -Xlinker -dependency_info -Xlinker /Users/chrishocking/Library/Developer/Xcode/DerivedData/Hammerspoon-eybhxyeverxtpibujwghieyonrun/Build/Intermediates.noindex/Hammerspoon.build/Release/application.build/Objects-normal/x86_64/application_dependency_info.dat -o /Users/chrishocking/Library/Developer/Xcode/DerivedData/Hammerspoon-eybhxyeverxtpibujwghieyonrun/Build/Products/Release/libapplication.dylib

I'll keep playing, but in the meantime any ideas?

@latenitefilms
Copy link
Contributor

Never mind... worked it out. Will test it out and let you know any problems I find in a new issue (to keep things clean).

@asmagill
Copy link
Member Author

asmagill commented Apr 8, 2020

Should the docs describing the changes to LuaSkin be added to the standard Hammerspoon wiki, or is there another place we should be using for LuaSkin specific documentation?

Should the warning printed when LuaSkin detects the old usage of [LuaSkin shared] include the URL?

The specific docs I'm talking about are:

And, any suggestions to make them better or clearer? Any other changes?

@cmsj
Copy link
Member

cmsj commented Apr 8, 2020

@asmagill I think the HS wiki is a good place for them, they're helpful docs that don't really fit in the API docs. Also having the URL in the warning is a great idea.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants