Automated build script

[sergeylukin]

Hi,

I've noticed the note in README file about you looking for automated build script so I thought maybe I can help with that. I've looked into .travis.yml file and as I understand you compile .smx files for each plugin in use and print the status in Gitter chat room from Travis. I haven't found any deployment stage there, please correct me if I missed it.

I assume you clone the repo / pull changes into production server after successful build manually and compile all .smx files over again in order to apply changes on production. Is my assumption correct? :)

I think what you need is automated deployment process which would automatically take pushed/merged changes to master branch and apply them on production server (only if Travis successfully compiled .smx files of course), is that what you're looking for?

I guess it's too early for me to suggest something as I need more details in order to come up with potentially solid solutions so I will be glad to collaborate on that once you decide to proceed. Feel free to close this issue if it's not relevant any more though.

Cheers

P.S. My nickname in ND is pogran_

[stickz]

Thanks for the reply! I am looking for a script to compile the .smx files and copy them to the updater section of the github. Let's use structminigame.sp as an perfect example. After it gets compiled to .smx , it would be amazing if this could be copied to the plugins folder.

The plugins folder for structminigame is located here.
https://github.com/stickz/Redstone/tree/master/updater/structminigame/plugins

If this is possible, it would fully automate the server update process. When a plugin is changed, simply merging the pull request and bumping the version would automatically update the server. Currently, I have to manually compile the .smx file and copy it to github which is time consuming.

How it works on the server side, is it reads this script right here.
https://github.com/stickz/Redstone/blob/master/updater/structminigame/structminigame.txt
When the version is higher than currently on the server, it will update the plugin and it's phrases.

[sergeylukin]

It's possible. I will submit a PR shortly. Stay tuned.

[stickz]

Thanks!

[sergeylukin]

Next improvement I'd suggest is either:

• committing and pushing compiled files from Travis CI to git repo, for that you'd need to setup encrypted value with github auth token in travis, see this for reference.
• adding compiled files to .gitignore and sending them over to production server by scp for example. This is usually preferred but maybe less suitable in your case because it would complicate the rollback process.

[stickz]

Yeah, I think pushing the compiled files from Travis to Git would be the best option. So, I need to store my public key in an encrypted variable for that method to work?

[sergeylukin]

Not exactly, public key is stored on host while client provides private key, so in order to send data via SSH Travis would need private key. Encrypting private key for Travis is not easy though, so I recommend storing OAuth token and pushing via HTTPS protocol instead. Here is what you can do:

• generate new token here

• install Ruby Travis CLI or NodeJS travis-encrypt

• encrypt generated token with travis encrypt -r stickz/Redstone GH_TOKEN=<secret token here> or travis-encrypt -r stickz/Redstone GH_TOKEN=<secret token here> for Ruby/NodeJS tool respectively.

• replace env directive with following in .travis.yml:

  env:
    global:
      - secure: "<encrypted string>"
    matrix:
      - SOURCEMOD=stable
      - SOURCEMOD=dev
  

• let me know once done and I will submit PR with implementation

[stickz]

Done.

[sergeylukin]

Great. We will also need an orphan branch that would contain all the files required by the server.
Because plugins compilation produces different .smx files every time and Travis is triggered every time master branch changes, we need to push to a different branch from Travis in order to prevent infinite loop here. This will also introduce a better separation of concerns at a later stage.

Please do following:

• Create an orphan branch from project root directory, I'll call it build for this example but feel free to find a more appropriate name:

  $ git checkout --orphan build && git rm -rf . && echo ".tmp/" > .gitignore
  $ git add . && git commit -m'Initial commit'
  $ git push origin build
  $ git checkout master
  

• Select "Build only if .travis.yml is present" in your Travis CI settings page. This will prevent Travis from doing anything when build branch is modified.

After we have the implementation up and running production server will be treated with the contents of build branch. I will make sure ./updater/ is pushed there but let me know if there are any other required directories.

[stickz]

The orphan branch has been created. And ./updater/ is the only required directory.

[sergeylukin]

Will submit implementation PR shortly. Stay tuned.

[sergeylukin]

Great, it's working as can be seen here https://travis-ci.org/stickz/Redstone/builds/113979798 just don't forget to select "Build only if .travis.yml is present" in your Travis CI settings page ;) Because otherwise it will try to build the build branch which is pointless and generates failed build for no reason...

[stickz]

Alright, I just enabled that option in my Travis CL settings. Everything is working perfectly. Thanks for the build script. This will speed up the server update process tremendously!

[sergeylukin]

Sure, glad I could help with that. Once you feel confident about the build branch and integrate it into production server all *.smx files can be removed from master branch and added to .gitignore.

One caveat that shouldn't be ignored from now on is because Travis may print the real value of encrypted GH_TOKEN environment variable if instructed so, it's important to not merge commits that contain something like echo $GH_TOKEN as otherwise your OAuth token will be logged in Travis job console and one could gain write access to your repo. Something to keep in mind.

I think we could improve the build process further by automating the "bump up" step. Feel free to share more details on minimum of what updater requires in ./updater/ directory in order to update the server and we may work out a solution that would automate that as well.

[stickz]

Using the structminigame example again, the server reads the .txt file in-order to update plugins. If the version on that file is higher than the current plugin value, it will update it.
https://github.com/stickz/Redstone/blob/master/updater/structminigame/structminigame.txt

"Version"
{
    "Latest"    "1.0.8"
}

Under the public Plugin:myinfo in the structminigame.sp there is a version value.
https://github.com/stickz/Redstone/blob/master/addons/sourcemod/scripting/structminigame.sp

#define VERSION "1.0.8"
public Plugin:myinfo =
{
    name = "[ND] Structure Killings",
    author = "databomb edited by stickz",
    description = "Provides a mini-game and announcement for structure killing",
    version = VERSION,
    url = "vintagejailbreak.org"
};

Updating line 7 on the .txt file to match the value of the constant or string after the version =, will fully automate the version bumping process. Above is one way to express the plugin version. The other way to express it is like this version="1.0.8".

[stickz]

Also, the translation updater is a special case which should automatically bump the version in the nd_translation_updater.sp file every time a /updater/nd_translation_update/translations/<name>.phrases.txt file is changed.

[sergeylukin]

I need following information in order to proceed:

• shouldn't structminigame bump up if any of it's translations files, say ./updater/structminigame/translations/structminigame.phrases.txt was changed?
• what are the valid formats for plugin version besides the currently used semver format? For example, in nd_translation_updater.sp, will unix timestamp format (for example: 1457246590) do the trick? Will version change from 1.0.9 to 1457246590 trigger the update?
• ./updater/{PLUGIN}/{PLUGIN}.txt files will not contain lines with "Notes", is it OK?
• Will ./updater/{PLUGIN}/{PLUGIN}.txt files be still valid if Tab spaces will be replaced with whitespace characters?
• What is the meaning of Previous property in ./updater/nd_translation_updater/nd_translation_updater.txt ? Is it required?
• What is the meaning of Patch directive in ./updater/nd_translation_updater/nd_translation_updater.txt ? Is it required?

Thanks

[stickz]

• I should mention that the updater plugin documentation is here.
• Yeah, it might be best to keep the translations separate, so, they only get updated when required. I guess anything should bump when phrases are changed, not just the translation updater.
• Here's the response from the plugin developer about the timestamps.
• The notes are pretty useless as everything is on github. That is fine.
• I'm almost sure the plugin simply looks for the quotation plus bracket syntax and doesn't really care about the type of spaces used.
• The previous andpatchdirectives simply reduce the amount of files which need to be downloaded to the server. If the version matches the previous one, it will only download what's inside patch; otherwise, it will download the entire list of things. Implementation of this would be nice; but; that's an optional feature which isn't required.

[sergeylukin]

That's perfect!

[sergeylukin]

Just logging in my findings for reference:

• Updater indeed simply compares value of version->latest from SMC file against value fetched via GetPluginInfo function for a specific plugin. Here is the exact line where it takes place: filesys.sp:156
  So no matter whether version value is incremented or decremented or changed to a random string value, it should update it as long as values are different.
• Regarding the whitespace characters, Updater is utilizing SourceMod's SMCParser to parse the files. Format is simply called "SourceMod configuration files", I couldn't find detailed spec so studied the Parser itself and here is what I found. ParseStream_SMC method inside TextParsers.cpp contains the core logic of how SMC files contents is parsed. Apparently whitespace characters are testified with g_ws_chartable[255] array which contains following characters: '\n', '\v', '\r', '\t', '\f', ' ' so I guess we can safely use regular whitespace characters instead of Tab spaces as they both exist in same characters group.

[sergeylukin]

I've noticed that some plugins from addons/sourcemod/scripting/ don't have corresponding directory with updater file and compiled plugin in updater/, is it on purpose? Automated mechanism will create directory in updater/ + updater file for all plugins found in addons/sourcemod/scripting/ equally, is that OK?

[stickz]

Those plugins do not have auto-updater support added to them. It's redundant having the automated mechanism create them. It should exclude or ignore plugins without a directory. This isn't something which should be automated, adding updater support should be done manually. There are many optimization and futuristic compatibility issues I could foresee if they are created automatically.

[sergeylukin]

Took that in count. Updater info is only created for plugins that have directories in ./updater

[stickz]

The sp file version needs changed or plugins will keep resending to the server in a loop.
For example on the structminigame plugin, the version need changed.

public Plugin:myinfo =
{
    name = "[ND] Structure Killings",
    author = "databomb edited by stickz",
    description = "Provides a mini-game and announcement for structure killing",
    version = "c966387",
    url = "vintagejailbreak.org"
};

Issue introduced on pull request. #47

[sergeylukin]

The version should stay consistent between compiled .smx and updater's .txt files so there is no reason why infinite loop in update mechanism would occur. Please let me know if that's not the case though.

[stickz]

I've investigated further, that works as intended without looping.

[sergeylukin]

Great. Now after we've finished implementing fully automated build mechanism (which was an interesting experience for me btw) we know better the requirements and can think about how to refactor things around in order to make implementation even better. I guess I will start discussion in separate thread and share my vision there while this issue can be closed. @stickz what do you think?

[stickz]

Sounds good!