CalDAV/CardDAV Testing Framework used by CalendarServer
Python Shell
Latest commit a90c967 Feb 20, 2017 @cyrusdaboo cyrusdaboo Fix git URL.

README.md

CALDAVTESTER

INTRODUCTION

CalDAVTester is a Python app that will run a series of scripted tests against a CalDAV server and verify the output, and optionally measure the time taken to complete one or more repeated requests. The tests are defined by XML files and ancillary HTTP request body files. A number of different verification options are provided.

Many tests are included in this package.

CalDAVTester can be extended to run tests against any type of HTTP server protocol by simply defining a new set of XML files.

INSTALL

Get CalDAVTester from github, create a virtualenv to run it in and install into the virtualenv:

git clone https://github.com/apple/ccs-caldavtester.git
virtualenv venv
source venv/bin/activate
pip install -r requirements.txt

COMMAND LINE OPTIONS

CalDAVTester is run via the testcaldav.py script:

testcaldav.py
	[-s filename]
	[-x dirpath]
	[--basedir dirpath]
	[--ssl]
	[--all]
	[--random]
	[--random-seed SEED]
	[--stop]
	[--print-details-onfail]
	[--always-print-request]
	[--always-print-response]
	[--exclude filename]
	[--observer OBSERVER]
	file1 file2 ...

-s : filename specifies the file to use for server information
(default is 'serverinfo.xml').

-x : directory path for test scripts
(default is 'scripts/tests').

--basedir : directory path for serverinfo.xml, test/ and data/,
	overrides -s and -x values

-p : filename specifies the file to use to populate the server with
data. Server data population only occurs when this option is
present.

-d : in conjunction with -p, if present specifies that the populated
data be removed after all tests have completed.

--ssl : run tests using SSL/https connections to the server.

--all : execute all tests found in the working directory. Each .xml
file in that directory is examined and those corresponding to the
caldavtest.dtd are executed.

--random : randomize the order in which the tests are run.

--random-seed SEED : a specific random seed to use.

--stop : stop running all tests after one test file fails.

--print-details-onfail : print HTTP request/response when a test fails.

--always-print-request : always print HTTP request.

--always-print-response : always print HTTP response.

--exclude FILE : when running with --all, exclude the file from the test run. 

--observer OBSEREVER : specify one or more times to change which classes are
used to process log and trace messages during a test. The OBSERVER name must
be the name of a module in the observers package. The default observer is the
"log" observer. Available observers are:

	"log" - produces an output similar to Python unit tests.
	"trace" - produces an output similar to the original output format.
	"loadfiles" - prints each test file as it is loaded.
	"jsondump" - prints a JSON representation of the test results.

file1 file2 ...: a list of test files to execute tests from.

QUICKSTART

Edit the serverinfo.xml file to run the test against your server setup.

Prior to running, make sure you are in the virtualenv:

source venv/bin/activate

Run testcaldav.py --all on the command line to run the tests. The app will print its progress through the tests.

EXECUTION PROCESS

  1. Read in XML config.
  2. Execute <start> requests.
  3. For each <test-suite>, run each <test> the specified number of times, executing each <request> in the test and verifying them.
  4. Delete any resources from requests marked with 'end-delete'.
  5. Execute <end> requests.

XML SCRIPT FILES

serverinfo.dtd

Defines the XML DTD for the server information XML file:

ELEMENT <host>
	host name for server to test.

ELEMENT <nonsslport>
	port to use to connect to server (non-SSL).

ELEMENT <sslport>
	port to use to connect to server (SSL).

ELEMENT <authtype>
	HTTP authentication method to use.

ELEMENT <certdir>
	Base directory for TLS client certs.

ELEMENT <waitcount>
	For requests that wait, defines how many iterations to wait for
	[Default: 120].

ELEMENT <waitdelay>
	For requests that wait, defines how long between iterations to
	wait for in seconds [Default: 0.25].

ELEMENT <waitsuccess>
	For requests with the wait-for-success options, defines how many
	seconds to wait [Default: 10].

ELEMENT <features>
	list of features for the server under test.

	ELEMENT <feature>
		specific feature supported by the server under test,
		used to do conditional testing.

ELEMENT <substitutions>
	used to encapsulate all variable substitutions.

	ELEMENT <substitution>
		a variable substitution - the repeat attribute can
		be used to repeat the substitution a set number of
		times whilst generating different substitutions.

		ELEMENT <key>
			the substitution key (usually '$xxx:').

		ELEMENT <value>
			the substitution value.

	ELEMENT <repeat>
		allow repeating substitutions for the specified count.

caldavtest.dtd:

Defines the XML DTD for test script files:

ATTRIBUTE ignore-all
	used on the top-level XML element to indicate whether this test
	is run when the --all command line switch for testcaldav.py is
	used. When set to 'no' the test is not run unless the file is
	explicitly specified on the command line.

ELEMENT <description>
	a description for this test script.

ELEMENT <require-feature>
	set of features.

	ELEMENT <feature>
		feature that server must support for this entire test
		script to run.

ELEMENT <exclude-feature>
	set of features.

	ELEMENT <feature>
		feature that server must not support for this entire test
		script to run.

ELEMENT <start>
	defines a series of requests that are executed before testing
	starts. This can be used to initialize a set of calendar
	resources on which tests can be run.

ELEMENT <end>
	defines a series of requests that are executed after testing is
	complete. This can be used to clean-up the server after testing.
	Note that there are special mechanisms in place to allow
	resources created during testing to be automatically deleted
	after testing, so there is no need to explicitly delete those
	resources here.

ELEMENT <test-suite>
	defines a group of tests to be run. The suite is given a name
	and has an 'ignore' attribute that can be used to disable it.

	ATTRIBUTE name
		name/description of test-suite.
	ATTRIBUTE ignore
		if set to 'yes' then the entire test-suite will be skipped.
	ATTRIBUTE only
		if set to 'yes' then all other test-suites (except others with
		the same attribute value set) will be skipped.

	ELEMENT <require-feature>
		set of features.

		ELEMENT <feature>
			feature that server must support for this test
			suite to run.

	ELEMENT <exclude-feature>
		set of features.

		ELEMENT <feature>
			feature that server must not support for this test
			suite to run.

ELEMENT <test>
	defines a single test within a test suite. A test has a name,
	description and one or more requests associated with it. There
	is also an 'ignore' attribute to disable the test. Tests can be
	executed multiple times by setting the 'count' attribute to a
	value greater than 1. Timing information about the test can be
	printed out by setting the 'stats' attribute to 'yes'.

	ATTRIBUTE name
		name of test.
	ATTRIBUTE count
		number of times to run the test. This allows tests to be
		easily repeated.
	ATTRIBUTE stats
		if set to 'yes' then timing information for the test will be
		printed.
	ATTRIBUTE ignore
		if set to 'yes' then the entire test will be skipped.

	ELEMENT <require-feature>
		set of features.

		ELEMENT <feature>
			feature that server must support for this test
			to run.

	ELEMENT <exclude-feature>
		set of features.

		ELEMENT <feature>
			feature that server must not support for this test
			to run.

ELEMENT <description>
	detailed description of the test.

ELEMENT <pause>
	halt tests and wait for user input. Useful for stopping tests to set a
	break point or examine server state, and then continue on.

ELEMENT <request>
	defines an HTTP request to send to the server. Attributes on the
	element are:

	ATTRIBUTE auth
		if 'yes', HTTP Basic authentication is done in the request.
	ATTRIBUTE user
		if provided this value is used as the user id for HTTP Basic
		authentication instead of the one in the serverinfo
		file.
	ATTRIBUTE pswd
		if provided this value is used as the password for HTTP
		Basic authentication instead of the one in the serverinfo
		file.
	ATTRIBUTE cert
		if provided this value is used as the file name for a TLS
		client certificate to be used with the request.
	ATTRIBUTE end-delete
		if set to 'yes', then the resource targeted by the request
		is deleted after testing is complete, but before the
		requests in the <end> element are run. This allows for quick
		clean-up of resources created during testing.
	ATTRIBUTE print-response
		if set to 'yes' then the HTTP response (header and body) is
		printed along with test results.
	ATTRIBUTE wait-for-success
		if set to 'yes' then the HTTP request will repeat over and over
		for a set amount of time waiting for the verifiers to pass. If
		time expires without success then the overall request fails. The
		length of time is controlled by the <waittime> element in the
		serverinfo file (defaults to 10 seconds).

	ELEMENT <require-feature>
		set of features.

		ELEMENT <feature>
			feature that server must support for this request
			to run.

	ELEMENT <exclude-feature>
		set of features.

		ELEMENT <feature>
			feature that server must not support for this request
			to run.

	ELEMENT <method>
		the HTTP method for this request. There are some 'special' methods that do some useful 'compound' operations:
			1) DELETEALL - deletes all resources within the collections specified by the <ruri> elements.
			2) DELAY - pause for the number of seconds specified by the <ruri> element.
			3) GETNEW - get the data from the newest resource in the collection specified by the <ruri> element and put its URI
					    into the $ variable for later use in an <ruri> element.
			4) WAITCOUNT N - wait until at least a certain number of resources "N" appear in a collection.
			5) WAITDELETEALL - wait until at least a certain number of resources appear in a collection, then delete all child
							   resources in that collection.
			6) GETCHANGED - the tool tracks the Etags on resources retrieved via GET. This special method will poll the specified
							resource until the Etag returned in the response is different from the one found in the most recent
							test.  
			6) GETOTHER - the tool finds the newest sibling resource to the one specified in the <ruri> element.  
			6) GETCONTAINS XXX - the tool finds the child resource whose content contains the supplied text "XXX".  

	ELEMENT <ruri>
		the URI of the request. Multiple <ruri>'s are allowed with DELETEALL only.
		The characters "**" may be used to cause a random uuid to be inserted where
		those two characters appear. The characters "##" may be used to insert the
		current test count iteration where those two characters occur.

	ELEMENT <header>
		can be used to specify additional headers in the request.

		ELEMENT <name>
			the header name.

		ELEMENT <value>
			the header value.

	ELEMENT <data>
		used to specify the source and nature of data used in the
		request body, if there is one.
		
		ATTRIBUTE substitutions
			if set to 'yes' then '$xxx:' style variable substitutions
			will be performed on the data before it is sent in the request.
		ATTRIBUTE generate
			if set to 'yes' then a basic calendar data "fuzzing" is done to
			the source data to make it unique and up to date.

		ELEMENT <content-type>
			the MIME content type for the request body.

		ELEMENT <filepath>
			the relative path for the file containing the request body
			data.

		ELEMENT <generator>
			a callback and set of arguments used to generate the data.
			
			ELEMENT <callback>
				the name of the generator method to execute.
	
			ELEMENT <arg>
				arguments sent to the generator method.
	
				ELEMENT <name>
					the name of the argument.
	
				ELEMENT <value>
					values for the argument.

		ELEMENT <substitute>
			a set of substitution variables to use on this data only.
		
			ELEMENT <name>
				the variable name.
	
			ELEMENT <value>
				the variable value.

	ELEMENT <verify>
		if present, used to specify a procedures for verifying that the
		request executed as expected.

		ELEMENT <require-feature>
			set of features.
	
			ELEMENT <feature>
				feature that server must support for this verification
				to be checked.

		ELEMENT <exclude-feature>
			set of features.

		ELEMENT <feature>
			feature that server must not support for this verification
			to be checked.
		
		ELEMENT <callback>
			the name of the verification method to execute.

		ELEMENT <arg>
			arguments sent to the verification method.

			ELEMENT <name>
				the name of the argument.

			ELEMENT <value>
				values for the argument.

    ELEMENT <graburi>
        if present, this stores the value of the actual request URI
        used in a named variable which can be used in subsequent requests.
        Useful for capturing URIs when the GETNEW method is used.
        
    ELEMENT <grabcount>
        if present, this stores the number of child responses in a
        {DAV:}multistatus response into the named variable which
        can be used in subsequent requests. This is useful for
        capturing the current count so that a change in the count
        can be tested for later.
        
	ELEMENT <grabheader>
		if present, this stores the value of the specified header
		returned in the response in a named variable which can be used
		in subsequent requests.
		
	ELEMENT <grabproperty>
		if present, this stores the value of the specified property
		returned in a PROPFIND response in a named variable which can
		be used in subsequent requests.
		
    ELEMENT <grabelement>
        if present, this stores the text representation of an XML
        element extracted from the response body in a named variable
        which can be used in subsequent requests.
        
    ELEMENT <grabjson>
        if present, this stores the text representation of a JSON
        object extracted from the response body in a named variable
        which can be used in subsequent requests.
        
	ELEMENT <grabcalproperty>
		if present, this stores a calendar property value in a named
		variable which can be used in subsequent request. The syntax for
		<name> element is component/propname (e.g. "VEVENT/SUMMARY").

	ELEMENT <grabcalparameter>
		if present, this stores a calendar parameter value in a named
		variable which can be used in subsequent request. The syntax for
		<name> element is component/propname/paramname$propvalue where
		the option $propvalue allows a specific property to be selected 
		(e.g. "VEVENT/DTSTART/TZID", or
		"VEVENT/ATTENDEE/PARTSTAT$mailto:user01@example.com").

VERIFICATION Methods

acltems

Performs a check of multi-status response body and checks to see whether the specified privileges are granted or denied on each resource in the response for the current user (i.e. tests the DAV:current-user-privilege-set).

Argument: 'granted'
	A set of privileges that must be granted.

Argument: 'denied'
	A set of privileges that must be denied denied.

Example:

<verify>
	<callback>multistatusitems</callback>
	<arg>
		<name>granted</name>
		<value>DAV:read</value>
	</arg>
	<arg>
		<name>denied</name>
		<value>DAV:write</value>
		<value>DAV:write-acl</value>
	</arg>
</verify>

calandarDataMatch

Similar to data match but tries to "normalize" the calendar data so that e.g., different ordering of properties is not significant.

Argument: 'filepath'
	The file path to a file containing data to match the response body to.

Example:

<verify>
	<callback>dataMatch</callback>
	<arg>
		<name>filepath</name>
		<value>resources/put.ics</value>
	</arg>
</verify>

dataMatch

Performs a check of response body and matches it against the data in the specified file.

Argument: 'filepath'
	The file path to a file containing data to match the response body to.

Example:

<verify>
	<callback>dataMatch</callback>
	<arg>
		<name>filepath</name>
		<value>resources/put.ics</value>
	</arg>
</verify>

dataString

Performs a check of response body tries to find occurrences of the specified strings or the absence of specified strings.

Argument: 'equals'
    One or more strings that must match exactly in the data (case-sensitive).

Argument: 'contains'
    One or more strings that must be contained in the data (case-sensitive).

Argument: 'notcontains'
	One or more strings that must not be contained in the data (case-sensitive).

Example:

<verify>
	<callback>dataString</callback>
	<arg>
		<name>contains</name>
		<value>BEGIN:VEVENT</value>
	</arg>
	<arg>
		<name>notcontains</name>
		<value>BEGIN:VTODO</value>
	</arg>
</verify>

freeBusy

Performs a check of the response body to verify it contains an iCalendar VFREEBUSY object with the specified busy periods and types.

Argument: 'busy'
	A set of iCalendar PERIOD values for FBTYPE=BUSY periods
	expected in the response.

Argument: 'tentative'
	A set of iCalendar PERIOD values for FBTYPE=BUSY-TENTATIVE
	periods expected in the response.

Argument: 'unavailable'
	A set of iCalendar PERIOD values for FBTYPE=BUSY-UNAVAILABLE
	periods expected in the response.

Argument: 'duration'
	If present the period values being checked use duration rather then
	end time.

Example:

<verify>
	<callback>freeBusy</callback>
	<arg>
		<name>busy</name>
		<value>20060107T010000Z/20060107T020000Z</value>
		<value>20060107T150000Z/20060107T163000Z</value>
		<value>20060108T150000Z/20060108T180000Z</value>
	</arg>
	<arg>
		<name>unavailable</name>
		<value>20060108T130000Z/20060108T150000Z</value>
	</arg>
	<arg>
		<name>tentative</name>
		<value>20060108T160000Z/20060108T170000Z</value>
		<value>20060108T210000Z/20060108T213000Z</value>
	</arg>
</verify>

header

Performs a check of response header and value. This can be used to test for the presence or absence of a header, or the presence of a header with a specific value.

Argument: 'header'
	This can be specified in one of three forms:
	
		'headername' - will test for the presence of the response
		header named 'header name'.

		'headername$value' - will test for the presence of the
		response header named 'headername' and also check that its
		value matches 'value'.

		'!headername' - will test for the absence of a header named
		'headername' in the response header.

Example:

<verify>
	<callback>header</callback>
	<arg>
		<name>header</name>
		<value>Content-type$text/plain</value>
	</arg>
</verify>

jcalDataMatch

Like calendarDataMatch except that comparison is done using jCal data.

jsonPointerMatch

Compares the response with a JSON pointer and returns TRUE if there is a match, otherwise False. The pointer is the absolute pointer from the root down. A JSON object's string value can be checked by append "~$" and the string value to test to the JSON pointer value. To test for a null value append "~~". A single "." can be used as a reference-token in the JSON pointer to match against any member or array item at that position in the document.

Argument: 'exists'
	JSON pointer for a JSON item to check the presence of
	in the response.

Argument: 'notexists'
	JSON pointer for a JSON item to check the absence of
	in the response.

Example:

<verify>
	<callback>jsonPointerMatch</callback>
	<arg>
		<name>exists</name>
		<value>/responses/response</value>
	</arg>
	<arg>
		<name>notexists</name>
		<value>/responses/response/name~$ABC</value>
	</arg>
	<arg>
		<name>exists</name>
		<value>/responses/./name~$XYZ</value>
	</arg>
</verify>

multistatusItems

Performs a check of multi-status response body and checks to see what hrefs were returned and whether those had a good (2xx) or bad (non-2xx) response code. The overall response status must be 207.

Argument: 'okhrefs'
	A set of hrefs for which a 2xx response status is required.

Argument: 'badhrefs'
	A set of hrefs for which a non-2xx response status is required.

Argument: 'prefix'
	A prefix that is appended to all of the specified okhrefs and
	badhrefs values.

Example:

<verify>
	<callback>multistatusitems</callback>
	<arg>
		<name>okhrefs</name>
		<value>/calendar/test/1.ics</value>
		<value>/calendar/test/2.ics</value>
		<value>/calendar/test/3.ics</value>
	</arg>
	<arg>
		<name>badhrefs</name>
		<value>/calendar/test/4.ics</value>
		<value>/calendar/test/5.ics</value>
		<value>/calendar/test/6.ics</value>
	</arg>
</verify>

postFreeBusy

Looks for specific FREEBUSY periods for a particular ATTENDEE.

Argument: 'attendee'
	Calendar user address for attendee to match.

Argument: 'busy'
	Period for FBTYPE=BUSY to match.

Argument: 'tentative'
	Period for FBTYPE=BUSY-TENTATIVE to match.

Argument: 'unavailable'
	Period for FBTYPE=BUSY-UNAVAILABLE to match.

Argument: 'events'
	Number of VEVENTs (in extended free-busy request) to match.

Example:

<verify>
	<callback>postFreeBusy</callback>
	<arg>
		<name>attendee</name>
		<value>$cuaddr1:</value>
	</arg>
	<arg>
		<name>busy</name>
		<value>20060101T230000Z/20060102T000000Z</value>
	</arg>
</verify>

prepostcondition

Performs a check of response body and status code to verify that a specific pre-/post-condition error was returned. The response status code has to be one of 403 or 409.

Argument: 'error'
	The expected XML element qualified-name to match.

Example:

<verify>
	<callback>prepostcondition</callback>
	<arg>
		<name>error</name>
		<value>DAV:too-many-matches</value>
	</arg>
</verify>

propfindItems

Performs a check of propfind multi-status response body and checks to see whether the returned properties (and optionally their values) are good (2xx) or bad (non-2xx) response code. The overall response status must be 207.

Argument: 'root-element'
	Expected root element for the XML response. Normally this is DAV:multistatus
	but, e.g., MKCOL ext uses a different root, but mostly looks like multistatus
	otherwise.

Argument: 'okprops'
	A set of properties for which a 2xx response status is required. Two forms can be used:
	
	'propname' - will test for the presence of the property named
	'propname'. The element data must be a qualified XML element
	name.

	'propname$value' - will test for the presence of the property
	named 'propname' and check that its value matches the provided
	'value'. The element data must be a qualified XML element name.
	XML elements in the property value can be tested provided proper
	XML escaping is used (see example).

	'propname!value' - will test for the presence of the property
	named 'propname' and check that its value does not match the provided
	'value'. The element data must be a qualified XML element name.
	XML elements in the property value can be tested provided proper
	XML escaping is used (see example).

Argument: 'badhrefs'
	A set of properties for which a non-2xx response status is
	required. The same two forms as used for 'okprops' can be used
	here.

Example:

<verify>
	<callback>propfindItems</callback>
	<arg>
		<name>okprops</name>
		<value>{DAV:}getetag</value>
		<value>{DAV:}getcontenttype$text/plain</value>
		<value>{X:}getstate$&lt;X:ok/&gt;</value>
	</arg>
	<arg>
		<name>badprops</name>
		<value>{X:}nostate</value>
	</arg>
</verify>

propfindValues

Performs a regular expression match against property values. The overall response status must be 207.

Argument: 'props'
	A set of properties for which a 2xx response status is required. Two forms can be used:
	
	'propname$value' - will test for property value match
	'propname!value' - will test for property value non-match

Argument: 'ignore'
	One or more href values for hrefs in the response which will be
	ignored. e.g. when doing a PROPFIND Depth:1, you may want to
	ignore the top-level resource when testing as only the
	properties on the child resources may be of interest.

Example:

<verify>
	<callback>propfindValues</callback>
	<arg>
		<name>props</name>
		<value>{DAV:}getcontenttype$text/.*</value>
		<value>{DAV:}getcontenttype!text/calendar</value>
	</arg>
	<arg>
		<name>ignore</name>
		<value>/calendars/test/</value>
	</arg>
</verify>

statusCode

Performs a simple test of the response status code and returns True if the code matches, otherwise False.

Argument: 'status'
	If the argument is not present, the any 2xx status code response
	will result in True. The status code value can be specified as
	'NNN' or 'Nxx' where 'N' is a digit and 'x' the letter x. In the
	later case, the verifier will return True if the response status
	code's 'major' digit matches the first digit.

Example:

<verify>
	<callback>statusCode</callback>
	<arg>
		<name>status</name>
		<value>2xx</value>
	</arg>
</verify>

xmlDataMatch

Compares the response with an XML data file and returns TRUE if there is a match, otherwise False.

Argument: 'filepath'
	The file path to a file containing data to match the response body to.

Argument: 'filter'
	Any specified XML elements will have their content removed from the
	response XML data before the comparison with the file data is done.
	This can be used to ignore element values that change in each request,
	e.g., a time stamp.

Example:

<verify>
	<callback>xmlDataMatch</callback>
	<arg>
		<name>filepath</name>
		<value>resource/test.xml</value>
	</arg>
	<arg>
		<name>filter</name>
		<value>{DAV:}getlastmodified</value>
	</arg>
</verify>

xmlElementMatch

Compares the response with an XML path and returns TRUE if there is a match, otherwise False. The path is the absolute xpath from the root element down. Attribute, attribute-value and text contents tests of the matched element can be done using:

[@attr] - "attr" is present as an attribute
[@attr=value] - "attr" is present as an attribute with the value "value"
[=text] - node text is "text".
[!text] - node text is not "text".
[*text] - node text contains "text".
[$text] - node text does not contain "text".
[+text] - node text starts with "text".
[^tag] - node has child element "tag".
[^tag=text] - node has child element "tag" with text "text".
[|] - node is empty.
[||] - node is not empty.
[json] - node contains valid JSON data.
[icalendar] - node contains valid iCalendare data.

Each path segment can now have its own test and "../" can be used to move up to
the parent. This allows testing for an element matching specific content plus
its sibling matching other specific content. e.g., "/{D}A/{D}B[=b]/../{D}C[=c]
which checks for an element {D}A with two child elements {D}B and {D}C each
with a specific value.

Argument: 'parent'
	ElementTree style path for an XML element to use as the root for any
	subsequent "exists" or "notexists" tests. This is useful for targeting
	a specific resource in a Depth:1 multistatus response.

Argument: 'exists'
	ElementTree style path for an XML element to check the presence of
	in the response.

Argument: 'notexists'
	ElementTree style path for an XML element to check the absence of
	in the response.

Example:

<verify>
	<callback>xmlDataMatch</callback>
	<arg>
		<name>exists</name>
		<value>{DAV:}response/{DAV:}href</value>
	</arg>
	<arg>
		<name>notexists</name>
		<value>{DAV:}response/{DAV:}getetag</value>
	</arg>
</verify>