C C++ Objective-C
Fetching latest commit…
Cannot retrieve the latest commit at this time.
|Failed to load latest commit information.|
A tool for managing platform-specific dependencies in a DVCS ============================================================ Although in most respects DVCSs are a big step forward from traditional centralised version control systems like Subversion and CVS, one thing they do poorly by comparison is managing platform-specific things like 3rd party libraries and tools. Adding things like that to version control is nice because it helps ensure that builds are repeatable. However people generally only want the copy of these things that is relevant to the platform they're developing on; if you check them all in to version control, then everyone gets all the copies for all platforms - which can waste a lot of space and bandwidth, as well as making repository operations take longer. The observation here is that, unlikes source code, for a platform-specific dependency (dep, from know on) you don't generally care about the version history: you just need to be able to get the right version for your code. So based on that, the idea behind this tool is to store the deps somewhere outside the version control system. We don't care where exactly, so long as (a) each dep is accessible through a URL; and (b) the contents of the URL doesn't change over time. This could be a subversion server; a regular webserver; or even a set of known locations on a shared file system. All we keep in our DVCS is a simple structured text file (format described below) which lists the URLS we depend on for each platform. We provide a tool which parses this file into a set of URLs that it then downloads. If the URL points at a zip file or tarball, we'll extract it automatically after downloading. Project structure ----------------- The project should have a root folder for all third party dependencies, which the DVCS is set to ignore. The dependencies will be unzipped in here after downloading. A good structure would look something like Project/ ... project specific stuff ... Thirdparty/ - root for all thirdparty stuff bin/ - for thirdparty executables include/ - include files for thirdparty libraries; this will be added to the compiler's include path. lib/ - thirdparty libraries; this will be added to the compiler's library search path. resources/ - other resources, such as test images, which shouldn't be stored in the DVCS. We don't mandate a specific structure though: this is a configuration management tool, not a build tool. So long as there is a location to extract stuff into, we're good. File format ----------- The file should sit at the root of the project and have a name ending in .deps. This isn't required: it'll be handled as the default, but it'll be possible to specify a different file on the command line. An example file might look like this: http://myserver/myproject/artwork-1.2.3.zip os "win": http://myserver/myproject/directx-wrapper-1.1.1.zip os "linux", "mac": http://myserver/myproject/opengl-wrapper-2.2.2.tar.gz The first line is a plain URL. Because it's not inside a conditional section, it will be downloaded for all platforms. The next line is a condition which checks if the 'os' variable is set to "win". If it is, the URL nested inside it will be downloaded as well. The next section is another condition, this time checking whether the 'os' variable is set to either "linux" or "mac"; if it has either of those values, the nested URL will be downloaded. The list of strings in the second condition is a literal string set. Variables also hold sets of strings, not just a single value. The condition is checking whether the intersection of the two sets is non-empty. If the intersection is non-empty then whatever's inside the condition is treated as if it appeared at the top level of the file. If the intersection is empty, whatever's inside the condition is ignored. The file uses indentation to show what's inside a condition. It doesn't matter how many spaces you indent by, so long as all the things inside the condition are indented by the same amount. We only recognise spaces as an indentation character; no tabs. It's possible to nest conditions inside each other. For example, if you need different copies of a library for 32 vs. 64 bit linux, you could write it like this: os "linux": bits "64": http://myserver/myproject/mylibrary-linux-64-1.2.3.tar.gz bits not "64": http://myserver/myproject/mylibrary-linux-any.3.2.1.tar.gz The 'bits not 64' line shows a slightly more complicated expression. You could also write the above as os "linux": bits = "64": http://myserver/myproject/mylibrary-linux-64-1.2.3.tar.gz bits != "64": http://myserver/myproject/mylibrary-linux-any.3.2.1.tar.gz Which will give identical results. It's possible to combine conditions using 'and' and 'or', with the usual semantics. For example both of the following are valid: os = "linux", "mac" and bits != "32" os "linux" or os "mac" and bits not "32" The 'and' and 'or' operators are left associative and have the same precedence, so will be evaluated in the order they're found. Simplified grammar for the file format -------------------------------------- Tokens are in capitals, literals are surrounded by quotes and grammar rules are in lower case. Other symbols are borrowed from regular expression syntax: parentheses to denote grouping, * for zero or more of the previous item, | meaning one or the other of the left and right values. The (simplified) grammar productions are: file ::= block block ::= (URL | conditional_section)* conditional_section ::= relation ((AND|OR) relation)* ':' INDENT block DEDENT relation ::= VAR ('not' | '=' | '!=')? str_value str_value ::= STR (',' STR)* The tokens are: URL = // the usual URL syntax. VAR = // any number of letters, digits, underscores and hyphens; no // spaces or other punctuation. INDENT = // When the parser detects a line indented relative to the one // above it. DEDENT = // When the parser detects the end of an indented section. STR = // String surrounded by double quotes. Some notes: - Whitespace is only significant at the start of a line and only in so far as it determines the level of indentation. - Tabs aren't allowed for indentation. Semantics of the file --------------------- The file describes a set of URLs to download. A URL at the top level is one which is always downloaded. A URL inside a conditional section is downloaded only if the condition is met. The resulting URLs are a set in the mathematical sense, so any duplicates will be ignored. A variable is an identifier for a set of strings. Variables are given values outside of the file (e.g. on the command line, or picked up from the runtime environment); there's no way to assign to them inside the file. Conditions consist of a variable name and a set of one or more values. The test is a set intersection: if the intersection is empty, the test fails; if it's non-empty it passes. In other words, if any of the values for a variable are in the list to the left of the variable name, the condition passes. If a condition passes, everything inside it is treated as if it appeared at the top level of the file. URLs are added to the download set and any nested conditions are evaluated as above. If a condition fails, we ignore everything nested inside it then carry on as normal after it. When we get to the end of the file, we've got a set of URLs. We download each of these in turn to either a default location, or a location specified on the command line. If the contents of any URL is a zip file, we unzip it after downloading it; ditto for tarballs. So where do we store the deps? ------------------------------ Anywhere! So long as they're accessible by a URL, it doesn't matter. It could very well be another version control system. SVN, for example, provides access to repository contents via a URL. Or equally it could just be a directory tree sitting behind a copy of apache. One thing that's important is that the contents of a URL don't change over time. Otherwise we'd get something unexpected when trying to rebuild an older version of our program. An easy way to do this is to make sure that the URL contains a version number; then a new version of the dependency will naturally have a different URL.