Skip to content

CollinScripts, Collins Actions, List Headers, oh my! #39

Closed
wants to merge 1,330 commits into from

4 participants

@ssalevan

Hey guys,
This change introduces several different notions into Collins:

CollinScripts:
A CollinScript is a piece of Scala that has the same level of access to Collins internals as code integrated directly into the Collins codebase. CollinScripts are loaded into a separate interpreter based upon Scala Script Engine:

http://code.google.com/p/scalascriptengine/

While SSE is relatively well unit tested, user configurability such as this can introduce instabilities into Collins, and so I've wrapped the engine in some try/catching and some logging, so if the script itself throws an exception, Collins itself will not go down. Most of this code can be found in the CollinScriptRegistry.scala file, particularly within the callMethod() function.

Collins Action:
A Collins Action is, simply, a user-definable action that can be executed upon an object from Collins, such as an Asset. Actions are implemented by way of Executors and Handlers:

Executor - Implements runCommand, runCommandString, runCommandBoolean from the ActionExecutor trait. An executor handles actually 'doing' the action, which is something like calling out to the shell or calling a method from a CollinScript.

Handler - A handler implements object-specific behavior based upon the interfaces specified in ActionExecutor

Collins Actions are configured by way of a util.config.ActionConfig object, and the handler for a Collins Action can be obtained from one of these objects by the following method call:

import collins.action.Action

Action.getMethod

where is currently one of the following:

AssetActionHandler
AssetsActionHandler (handles Page[AssetView])
CallbackActionHandler (handles PropertyChangeEvents)

Tag Decoration Changes:

Tag decorators have been tricked out a bit to support default values, column headers, and three concepts based upon Collins Actions: showIf, decoratorAction, and formatterAction. Here's an example:

HOSTNAME {
default="None"
header="Hostname"
showIf {
type=script
command=[
"collins.script.ShowIf.showHostname"
]
}
decoratorAction {
type=script
command=[
"collins.script.TagDecorators.decorateHostname"
]
}
formatterAction {
type=script
command=[
"collins.script.Formatters.ellipsis"
]
}
}

showIf is a Collins Action which is passed a PageAssetView or templates a command based upon methods from the Page[AssetView] object (if a shell exec) and determines a boolean true or false value based upon whether the tag should be shown for the current assets on the page.

decoratorAction is a Collins Action which is passed an AssetView (if a script) or templates a command based upon methods from the AssetView object (if a shell exec) and determines a String representation to assign to the tag. In the case above, we call out to a CollinScript and, when rendering the HOSTNAME pseudo-tag for an asset, pass the AssetView corresponding to it to the CollinScript, which proceeds to render it as a String.

formatterAction takes the output from the metadata tag or decoratorAction as a String and passes it either to the shell or to a CollinScript, where it is processed and formatted into human-readable form.

List Headers:

List Headers were previously hardcoded, and this change introduces the ability to set defaults based upon either asset metadata tags or pseudo-tags like the above HOSTNAME. Configuration for this feature can be found under conf/reference/list_reference.conf

Let me know what you think, and I very much appreciate the feedback!

Sussudio.

DanSimon and others added some commits Aug 15, 2012
@DanSimon DanSimon replacing Asset.find with Solr searching 65d3dd6
@DanSimon DanSimon fixing AssetSearchParameters solr serialization 363c87b
@DanSimon DanSimon adding sort field to routes, totally encapsulated solr key lookup a44f2eb
@DanSimon DanSimon allowing key aliases for api support, working on unit tests passing 03e42df
@DanSimon DanSimon solr range, date fixes 41b797e
@DanSimon DanSimon quoting string values, fixing date ranges 7a4e208
@DanSimon DanSimon new version router for routing based on API version 7419dc1
@DanSimon DanSimon adding support for default version routing ca0e852
@DanSimon DanSimon switching router to use ADT, partial functions 6ee5547
@DanSimon DanSimon adding version routing to get/assets 1e10a26
@DanSimon DanSimon switching from PartialFunction to Function1 for exhaustive matches 253704c
@DanSimon DanSimon fixing tests e5cef3a
@DanSimon DanSimon fixing bugs with empty queries ab6c8c5
@DanSimon DanSimon duh, don't search solr to re-index solr 69af7ac
@bmatheny bmatheny Ugh, need to reset these creds now 49b5c27
@bmatheny bmatheny Not sure how this has ever worked f9d0183
@bmatheny bmatheny Adding solrdata to gitignore 33b9f41
@bmatheny bmatheny Fixed comma separated headers and added API support to client 01f7703
@bmatheny bmatheny Started namespace cleanup and config refactor. Need to fix tests afte…
…r move
c842990
@DanSimon DanSimon removing version routing from asset find api
collins determines which search style to use based only on whether the
"query" parameter is empty or not
deac2b2
@DanSimon DanSimon cleaning up error handling of invalid versions b1ddb53
@bmatheny bmatheny Play upgrade, dynamic detection of config classes 0f21ef5
@bmatheny bmatheny Support for delaying initial file watch trigger f0605ac
@bmatheny bmatheny Bit of logging f635563
@DanSimon DanSimon fixing sorting in the web view 8ba879e
@DanSimon DanSimon converting sort direction from enum to ADT ba9f28f
@DanSimon DanSimon polymorphic PageParams constructor, type fixes 0daf8cf
@DanSimon DanSimon case insensitive sort parsing, fix tests 4a7d521
@bmatheny bmatheny Initial version of change detection for configs 9eaaa72
@DanSimon DanSimon sort direction toggling working 850c2e3
@bmatheny bmatheny Deleted Feature and refactored 75ec8de
@bmatheny bmatheny Power management uses Configurable b24f7d1
@bmatheny bmatheny Deleted Config class, started refactoring ac8a7f6
@bmatheny bmatheny Finished first batch of fixes, next one coming f63308b
@bmatheny bmatheny Code now cleanly compiles, now to fix tests ed9abc4
@bmatheny bmatheny Tests now compile, now to get them to pass 3376a06
@bmatheny bmatheny All tests passing a7c245f
@bmatheny bmatheny Bit of cleanup 2ca4341
@bmatheny bmatheny Started killing refs to Configuration d1fa618
@bmatheny bmatheny Renaming internal Configuration classes to Config 9d92d45
@bmatheny bmatheny Fixed tests 2acc622
@bmatheny bmatheny Hide dirty usage of Typesafe/Play config 74ef9be
@bmatheny bmatheny Moved decorator code to use new configs b3c310c
@bmatheny bmatheny Minor fixes for prod 7ea119b
@bmatheny bmatheny Stop using FileWatcher in ConfigWatch e8ef740
@bmatheny bmatheny Cleaned up remaining auth cruft aa75830
@bmatheny bmatheny Moved same config stuff into dev_base 97ca587
@bmatheny bmatheny Runs in production mode as well now 5ad6743
@bmatheny bmatheny Need permissions to be defs in case they change e85e782
@bmatheny bmatheny Cleaned up reference configs a bit 9c6a497
@bmatheny bmatheny Moved solr plugin to use new config stuff d20f1b0
@bmatheny bmatheny Remove some single use methods 368a42a
@bmatheny bmatheny Moved callback plugin into main source 150597f
@bmatheny bmatheny Allow callback commands to be specified as lists or strings
Scala sys.Process uses ProcessBuilder under the hood which just breaks up
commands and args by whitespace. If the command isn't something like 'cmd
--arg1=val1 --arg2=val2' where splitting on whitespace results in a valid
command, such as for printf (which may have a whitespace separated format
specifier) it is useful to allow the command to be explicitly specified as a
list.
b747db8
@bmatheny bmatheny Version bump to 0.2.2 a67d849
@bmatheny bmatheny Make sure plugins can work with the config system ff55d62
@bmatheny bmatheny CallbackManagerPlugin needs pluginInitialize as well f258db6
@bmatheny bmatheny Cleaning up some namespacing 006ff13
@bmatheny bmatheny Moving file into correct package path 6ae15e7
@bmatheny bmatheny Broke solr classes out into files 663018a
@bmatheny bmatheny Upgrading to play 2.0.3 6cf6885
@bmatheny bmatheny SolrPlugin looks like it isn't draining, I'm wondering if removeAll i…
…s working like I think
afeb410
@bmatheny bmatheny Updating README to point at play 2.0.3 905bef4
@bmatheny bmatheny Bug fixes
 * If the user specifies a callback command as a string, pass it to Process as a string
 * Asset doesn't work in a set, fixes solr indexing going crazy
0a44c08
@bmatheny bmatheny Assets can come back to life and should be indexed. f2e6d51
@bmatheny bmatheny Heap dump if OOM b551fa0
@bmatheny bmatheny Closes #3 - try and log runtime errors against an appropriate asset 2946606
@bmatheny bmatheny Moved common reflection stuff to MethodInvoker 21e9dac
@bmatheny bmatheny Pulled common reflection code from callback package into reflection p…
…ackage. Closes #8
9112363
@bmatheny bmatheny Resolving merge conflicts after config refactor. Code compiles, tests…
… still pass.
cdf668e
@bmatheny bmatheny Pulled Cache plugin into collins package ae9c2c2
@bmatheny bmatheny Using java.lang.Integer so we don't end up with an int 21e598c
@bmatheny bmatheny Moved permissions code into collins package df363ea
@bmatheny bmatheny Getting ready to throw out file watcher in favor of just using the gu…
…ava cache

loader
c9e901d
@bmatheny bmatheny Killed old FileWatcher code in favor of using a guava loading cache. …
…More

predictable behavior in failures.
69b65c4
@bmatheny bmatheny Moved provisioning code into collins package 60d4599
@DanSimon DanSimon adding new frontend query logging to capture search querystrings 77198d2
@bmatheny bmatheny Cleaned up provisioning plugin
Note that the custom build of snakeyaml is needed until
http://code.google.com/p/snakeyaml/issues/detail?id=156 is incorporated into a
release.
12fa0fe
@bmatheny bmatheny Adding official support (upstream) for the snakeyaml patch I submitted cd4ddc9
@bmatheny bmatheny Cleaned up configs a bit 764ed86
@bmatheny bmatheny Merge pull request #10 from tumblr/api-query-logging
adding new frontend query logging to capture search querystrings
c28d512
@bmatheny bmatheny Pulled remaining plugins into collins package 771df30
@bmatheny bmatheny Merge branch 'master' into plugin_cleanup d8dd43f
@bmatheny bmatheny Moved Provisioner and SoftLayer util plugins into util.plugins 21500da
@bmatheny bmatheny Finished plugin renaming. Will also want to move Solr/Ipmi plugins af…
…ter Dan's changes get merged
70ab147
@bmatheny bmatheny Log to STDOUT and MULTIPLEX 3fc9a70
@bmatheny bmatheny Cleaned up reference/dev configs 754682a
@bmatheny

One issue I discovered too late is that since this uses debug, we need to either change that to logger.info or update the logger config. Ready to go once that's done.

ok, I switched it to use logger.info instead.

Deploying now. Thanks Dan!

bmatheny and others added some commits Sep 6, 2012
@bmatheny bmatheny Equals, not colon 1dc142b
@DanSimon DanSimon frontend query logging uses info level instead of debug f44f0e1
Steve Salevan Adding the first, mostly working take on integrating a scripting engi…
…ne into Collins
00c7b33
@DanSimon DanSimon creating some tests to verify results of find requests 02879cd
@DanSimon DanSimon working on find tests 0bd8d98
@bmatheny bmatheny Prepping code for supporting reintake re #12
Fixed a couple of bugs:

  * Reloading a JProfile in dev was broken due to different classloaders
  * Fixed a bug in auth where upper case group names weren't properly handled

Also added support for features.sloppyTags, which allows any tags to be
completely managed outside of collins. Things in general will not work well by
doing this but can be useful for testing or one off situations.
a9fc911
@DanSimon DanSimon filling in tests 0b885e5
Steve Salevan A few more iterations towards working 8ade9ea
@bmatheny bmatheny Stop doing adhoc JSON formatting 85384bd
@bmatheny bmatheny Removing empty file 87c80d8
@bmatheny bmatheny Only allow SL server assets to be activated. Closes #13 da5d554
@bmatheny bmatheny Implemented delete support, closes #4
Only works on assets in Unallocated, Maintenance, Cancelled, Decommissioned
states.

Requires you to have controllers.AssetWebApi.deleteAsset permissions.

Not supported via API, web interface only for now.
aaab699
@bmatheny

These look like they may have been left in for debugging? Should these be error level logs?

@bmatheny

For fetching the current classloader you really want to make sure by default you use the play one (which you are above) so that class reloading works properly in dev. The way we do this elsewhere is something like:

val classLoader = try {
    import play.api.Play.current
    current.classloader
} catch {
    case e =>
        classOf[Asset].getClassLoader....
}
@bmatheny

Is myAss the default we want to use?

@bmatheny

I'm assuming the following two methods are for testing and were left in. We should probably remove these.

@bmatheny

Did you intend to change the namespace/referenceConfigFilename along with the class name?

@ssalevan

Hey Blake,
I apologize for the general disarray of the code as it stands; I discovered an issue upon further testing last night wherein scripts loaded by the script engine would resolve Collins namespace classes as their arguments but the classes would not match those currently loaded by the Play classloader, thereby resulting in a NoSuchMethodException upon invocation. I apologize that some of the more colorful pieces of that debug session have made it into the review, and I hope to have this issue sussed and all remaining points addressed by EOD.

@ssalevan

Very good news: your Play classpath suggestion resolved this issue, thank you for the awesome as always. I believe we're now on track for that EOD target, so keep a look out later on today for the next iteration. Thanks much!

@bmatheny

No worries. If they weren't occasionally useful we wouldn't do code reviews :)

bmatheny and others added some commits Sep 15, 2012
@bmatheny bmatheny General status model related cleanup
  * Removed Status.Enum - refs #25
  * Made asset update API endpoint smarter - refs #12
6338c66
@bmatheny bmatheny collins-shell support for state management
Found two nice little bugs in the process. partialUpdate was causing duplicate
callbacks to be triggered. Also found a nice bug where cache keys using the
oldValue from an update wouldn't get invalidated.
37bd341
@bmatheny bmatheny Removed duplicate functionality 2ed7f95
@bmatheny bmatheny Version bump to 0.2.5 b76941e
@bmatheny bmatheny Version bump to 0.2.8 f08d1f9
@bmatheny bmatheny Version bump to 0.2.9 ecaee72
@bmatheny bmatheny Version bump to 0.2.13 8f89464
@bmatheny bmatheny Gem upgrades for collins-client, state and shell 9a5a2f4
@bmatheny bmatheny Merged master 8a26e5c
@bmatheny bmatheny Merging from master 7d8165b
@bmatheny bmatheny Merging master 70010d1
Steve Salevan Fixing partial compilation/classpath linking bug 97d59c4
Steve Salevan Adding further code review improvements 2fe07ec
Steve Salevan Merge branch 'steve-configurable-headers' of github.com:tumblr/collin…
…s into steve-configurable-headers
0b67211
@DanSimon DanSimon adding ping to client, moving checksums to separate file 29d1868
@DanSimon DanSimon fixing up finder tests bad8f76
@bmatheny bmatheny Route docs and grouping c4a9440
@DanSimon DanSimon minor format fixes d8d2700
@DanSimon DanSimon Merge branch 'dan-all-solr-searches' of github.com:tumblr/collins int…
…o dan-all-solr-searches
b260d58
@DanSimon DanSimon Merge branch 'master' into finder-tests
Conflicts:
	app/controllers/Api.scala
f575ec5
@DanSimon DanSimon updating asset state tests to use integration module 494b811
@DanSimon DanSimon Merge branch 'master' into dan-all-solr-searches 1fb4791
@DanSimon DanSimon Merge branch 'finder-tests' into dan-all-solr-searches
Conflicts:
	support/ruby/collins-client/lib/collins/api.rb
fd9d042
@DanSimon DanSimon fixing some merge conflicts, tests bec6210
@DanSimon DanSimon fixing up tests e5771d2
Steve Salevan Tuning configuration system, making refresh thread-safer 8cdc3bf
Steve Salevan Removing detritus from main.scala.html 86d1362
Steve Salevan Adding a writelock to the tryRefresh method, as SSE itself isn't thre…
…adsafe and that gives me the willies
7866df5
@DanSimon DanSimon using SolrKeyResolver for static keys, fixing tests 7dc1d25
Steve Salevan Adding asset row class action for search results 9d6de0c
@bmatheny bmatheny Updated gitignore to reflect new gh-pages docs c148103
@DanSimon DanSimon conf fix 9fd6ab5
Steve Salevan Tuning SearchResults row class stuff 87a45d5
Steve Salevan Tuning SearchResults row class stuff 021fc13
Steve Salevan Tuning SearchResults row class stuff 0436fb6
@DanSimon DanSimon new util script, output cleanup b208372
@DanSimon DanSimon conf fix 3093aab
@DanSimon DanSimon GET /api/admin/solr returns 501 instead of 500 if solr not enabled 604ec80
@DanSimon DanSimon new StringValueFormat to handle regex/wildcard formatting c707dd1
@bmatheny bmatheny Merge pull request #42 from tumblr/finder-tests
New integration testing framework
624aa78
@bmatheny bmatheny Fixed conflict 3db2d22
@bmatheny bmatheny Updating gitignore b420c58
@bmatheny bmatheny Making solr on by default, enabled always true 3bcf978
@DanSimon DanSimon fixing more quoting and range value issues, moar tests 14c742e
@DanSimon DanSimon Merge branch 'dan-all-solr-searches' of github.com:tumblr/collins int…
…o dan-all-solr-searches
13b1810
@DanSimon DanSimon adding some more rpsec tests for regex 72a72fd
@DanSimon DanSimon fixing some asset finder behavior, tests 47b14f4
@bmatheny bmatheny Code now compiles with AssetType.Enum removed 6688704
@bmatheny bmatheny Tests now compile 853f656
@bmatheny bmatheny Added label column to AssetType to make it behave
Previously we used the Enum to resolve the names but the actual DB stored names
were basically only used for display. Moved the former db names into the label
column and made the name column the former enum string values.
4ebc151
@bmatheny bmatheny Use labels for asset types when we have them c02581b
@bmatheny bmatheny Merge pull request #1 from tumblr/dan-all-solr-searches
Replacing MySQL with Solr for all Asset Searching
876ba13
@bmatheny bmatheny Removing stray println ac04558
Steve Salevan Adding suggestions from Typesafe's James Roper which fix CollinScript…
… behavior

under dev mode
264c1e6
Steve Salevan Adding iterative polymorphic method match check as suggested by Typesafe a0b4b0e
@bmatheny bmatheny Adding new collins instance for EWR01 60ebd8b
@DanSimon DanSimon accounting for .* leading and trailing regex to wildcard conversion 63b50cb
@DanSimon DanSimon Merge branch 'master' of github.com:tumblr/collins cae3043
@bmatheny bmatheny Fixing merge conflict 53287a3
@DanSimon DanSimon fixing string fields to be case insensitive 6a54e59
@bmatheny bmatheny Removing some withName enumeration lookups a38dafc
@DanSimon DanSimon Merge branch 'master' of github.com:tumblr/collins ec9ab53
@bmatheny bmatheny Version bump to 0.2.6 9ecc679
@bmatheny bmatheny Updating new gem 0c73376
@bmatheny bmatheny Graph plugin support refs #41
Added support for showing graphs in the asset details view. Currently only
support for fibr graphs exist which won't generally be useful to anyone outside
of Tumblr. Creating support for a different graph system is pretty trivial so
hopefully there will be some community support.

Need to only show the tab pane if the plugin is enabled. Also make sure that the
config is reloadable.
7a5f41e
@bmatheny bmatheny Only show tab-pane if graph plugin is enabled 44d59bd
@bmatheny bmatheny Broke out FibrGraphConfig into own file a51a994
@bmatheny bmatheny Allow online graph config changes refs #41
Also fixed a bug where multiple solr populate operations could happen at the
same time.
a809344
@bmatheny bmatheny Merge branch 'master' into graph_plugin 9001f1f
@bmatheny bmatheny Log out of range exception 86e3900
@bmatheny bmatheny Encrypted tags now get quiet logging, closes #37
If values that are encrypted are changed, we just log the fact that they were
changed but not what the previous or new values are. This is a better
alternative than setting them to noLog
a8a617b
@bmatheny bmatheny Include previous/new status/state in log when values are changed fcfe773
@bmatheny bmatheny Ditching old collins servers ca02d34
Steve Salevan CollinScript cleanup ceaa0a4
Steve Salevan Merge branch 'master' into steve-configurable-headers
Conflicts:
	app/views/asset/list.scala.html
	conf/play.plugins
	conf/validations.conf
55e7c0c
Steve Salevan Refactoring, cleanup, and slightly increased concision e4993d4
Steve Salevan Merging in tag sort stuff eaf10f5
Steve Salevan Making header sorting work again d5062e0
Steve Salevan A little cleanup around refresh locking 8f80413
@ssalevan

Hey guys,
I think that CollinScript is in a state ready for the next code review; I've looped in some of the more recent changes, such as sortable tags and row coloration, and I've refactored the code a little bit for readability, concision and speed. Let me know what you think, and I very much appreciate the consideration!

@bmatheny bmatheny and 1 other commented on an outdated diff Sep 25, 2012
conf/reference/tagdecorators_reference.conf
decorators {
- PRIMARY_ROLE.decorator = ${tagdecorators.templates.search}
- SECONDARY_ROLE.decorator = ${tagdecorators.templates.search}
- POOL.decorator = ${tagdecorators.templates.search}
- GRAPHS {
- decorator = "<img src=\"{value}\">"
+ STATS_LINKS {
@bmatheny
bmatheny added a note Sep 25, 2012

This is confusing to me. Do the old style decorators still work?

@ssalevan
ssalevan added a note Sep 25, 2012

Dag, good catch. The old-style decorators most definitely still work, these musta gotten killed in a merge somewhere back. Restored the original configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@bmatheny bmatheny and 1 other commented on an outdated diff Sep 25, 2012
conf/collinscript.conf
@@ -0,0 +1,6 @@
@bmatheny
bmatheny added a note Sep 25, 2012

This doesn't override anything so does it need to exist?

@ssalevan
ssalevan added a note Sep 25, 2012

Deleted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@bmatheny bmatheny and 1 other commented on an outdated diff Sep 25, 2012
app/util/views/SearchResultsConfig.scala
@@ -0,0 +1,29 @@
+package util
+package views
+
+import config.Configurable
+import util.config.ActionConfig
+
+
+object SearchResultsConfig extends Configurable {
+
+ override val namespace = "searchresults"
+ override val referenceConfigFilename = "searchresults_reference.conf"
+
+ def defaultTagOrder = {
@bmatheny
bmatheny added a note Sep 25, 2012

I think you actually want stringSet, which coincidentally takes as a second argument the set to use if the config set is empty.

@ssalevan
ssalevan added a note Sep 25, 2012

Alas, we need the ordering mechanism of a list, which was why I went with that hacky route. I've implemented a similar default-accepting method in getStringList and have updated this config object to use it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@Primer42
Tumblr member

Does anyone know if this pull request is still current? Is it worth trying to sort out all of the merge conflicts, or is it a lost cause?

@Primer42 Primer42 closed this Aug 13, 2014
@maddalab maddalab deleted the steve-configurable-headers branch Jun 13, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.