Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarify User vs Builtin rules #574

Closed
ndmitchell opened this issue Mar 26, 2018 · 9 comments

Comments

Projects
None yet
2 participants
@ndmitchell
Copy link
Owner

commented Mar 26, 2018

See https://neilmitchell.blogspot.co.uk/2017/09/shake-016-revised-rule-definitions.html?showComment=1521767681664#c1857241005456990857 - concretely:

I really don't understand what what builtin- and user rules are.
What should I pass to 'addUserRule', what is the 'a' supposed to be?
What is the idea behind UserRule?
And what are 'builtin rules'?
When do I need to add them, and why are they called 'built-in', if users can/have to add them ;) ?

Very good point, Builtin rules aren't built in.

@sheyll

This comment has been minimized.

Copy link
Contributor

commented Mar 26, 2018

Thank you for creating the issue :)
I will try to figure out howto create a custom rule, and if I succeed I might post an example.

@ndmitchell

This comment has been minimized.

Copy link
Owner Author

commented Mar 26, 2018

That would be great. Are you able to say what the custom rule is? Generally I find it's quite rare that people actually need to use custom rules - often an oracle would be easier.

@sheyll

This comment has been minimized.

Copy link
Contributor

commented Mar 27, 2018

I don't know if an oracle would be easier, because I am not really sure what the exact differences to using builtin- and user-rules are.

I use shake as part of our internal build process, that not only compiles actual sources, but also generates VM images using a small library that wraps around command line tools like virsh and qemu-img. This library (that comes with a commandline executable) is called b9.

That library has an internal image repository mechanism, such that one b9 project could create a VM image that is stored in the repository und other b9 projects can use it.

The point of this repository is to reduce build time and disk space requirements.

Build time is reduced by managing concurrent access to VM-images and serializing image creation and consumption, also build machines can share their repositories over the network (internally via rsync/ssh). Sharing images reduces build time because often it is faster to transfer VM-images than to rebuild them locally.

Disk space requirements are reduced by deleting obsolete versions of these VM images.

The Shake glue code I need, would lookup the exact version (aka build-id) of one kind of image using the B9 library modules to access the repository. The key is just a an image name and the value is some kind of random looking string that is uniquely generated during a build.

This is example code, showing how to find out if a shared image idendified by a shared image name exists and if so, what build-id it has:

import qualified B9

(stuff omitted ...)

lookupCachedSharedImage
  :: MonadIO m => B9.SharedImageName -> m (Maybe B9.SharedImageBuildId)
lookupCachedSharedImage n = do
  cfgParser <- B9.readB9Config Nothing
  liftIO $ B9.run cfgParser B9.defaultB9Config go
 where
  go =
    extractNewestImageFromResults
      <$> B9.lookupSharedImages isAvailableOnLocalHost hasTheDesiredName
   where
    extractNewestImageFromResults =
      listToMaybe . map getBuildId . take 1 . reverse . map snd
      where
        getBuildId (B9.SharedImage _ _ i _ _) = i
    isAvailableOnLocalHost = (B9.Cache ==)
    hasTheDesiredName (B9.SharedImage n' _ _ _ _) = n == n'
@ndmitchell

This comment has been minimized.

Copy link
Owner Author

commented Apr 4, 2018

I just released shake 0.16.4 which provides newOracleCache. With that you can define a rule that maps between a SharedImageName and a SharedImageBuildId - looking it up if it exists, and building it otherwise. I'm going to claim (but am not yet convinced myself yet...) that all rules could be defined on top of newOracleCache, and that the builtin rule mechanism is just a way of making things more efficient (but unlikely to matter in most cases).

Can I suggest you try newOracleCache and report bugs if it is insufficient?

@sheyll

This comment has been minimized.

Copy link
Contributor

commented Apr 5, 2018

I just finished up a version with an integration based on 'addUserRule' and 'addBuiltinRule', see B9.Shake.SharedImageRules.

Could you please have a look at the use of 'addBuiltInRule', did I get the idea correctly?

I would proceed to write up a small blog article, using this or the or another planned Shake integration for B9 I need to add, as an example of how to develop custom shake stuff.
I could also try 'newOracleCache' for that next integration task.

I'll see to it on the weekend, I have what I need for work now.

@ndmitchell

This comment has been minimized.

@sheyll

This comment has been minimized.

Copy link
Contributor

commented Apr 13, 2018

Yeah, that was very helpful thanks!

@ndmitchell

This comment has been minimized.

Copy link
Owner Author

commented Apr 29, 2018

I wrote a lot of details at https://github.com/ndmitchell/shake/blob/master/src/Development/Shake/Rule.hs - does that clarify things? To see more completely, check out the repo, generate Haddock, and browse to Development.Shake.Rule.

@ndmitchell

This comment has been minimized.

Copy link
Owner Author

commented May 2, 2018

Treating this as closed, but please reopen if there are further clarifications required.

@ndmitchell ndmitchell closed this May 2, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.