RESTest: Black-Box Constraint-Based Testing of RESTful Web APIs - Supplementary material

This repository contains the supplementary material of the paper entitled "RESTest: Black-Box Constraint-Based Testing of RESTful Web APIs", submitted to ICSOC 2020. The data is divided into four categories:

1. Tools. Here we include links to the tools used in our work.
2. OAS specifications. List of OAS specifications used in our work (13 in total).
3. Evaluation data. This includes all output data generated by RESTest for the evaluation performed, including test cases, test results and statistics.
4. Bugs found. For those test cases that failed, we analysed the test results and tried to infer possible faults in the services causing the failures.

1. Tools

We have made used of the following three tools for the implementation of our constraint-based testing approach for RESTful APIs:

• RESTest. Framework for automated black-box testing of RESTful web APIs.
• IDL. Domain-specific language for the formal specification of inter-parameter dependencies in web APIs.
• IDLReasoner. Java library for the automated analysis of inter-parameter dependencies through constraint-programming.

2. OAS specifications

For the evaluation of RESTest, we tested 9 services from 6 commercial APIs. Since RESTest requires only the OAS specification of the API as input, we include here the specifications used in the evaluation. Note that 13 specifications were used in total, since we had to perform additional experiments for those services whose dependencies described in the API documentation portal did not match those actually implemented (4 out of 9 services).

• Foursquare - Search venues. OAS specification. Link to the API documentation.
• Foursquare - Search venues * (dependencies modified). OAS specification. Link to the API documentation.
• GitHub - Get user repos. OAS specification. Link to the API documentation.
• Stripe - Create coupon. OAS specification. Link to the API documentation.
• Stripe - Create coupon * (dependencies modified). OAS specification. Link to the API documentation.
• Stripe - Create product. OAS specification. Link to the API documentation.
• Stripe - Create product * (dependencies modified). OAS specification. Link to the API documentation.
• Tumblr - Get blog likes. OAS specification. Link to the API documentation.
• Yelp - Search businesses. OAS specification. Link to the API documentation.
• YouTube - Get comment threads. OAS specification. Link to the API documentation.
• YouTube - Get videos. OAS specification. Link to the API documentation.
• YouTube - Search. OAS specification. Link to the API documentation.
• YouTube - Search * (dependencies modified). OAS specification. Link to the API documentation.

3. Evaluation data

We performed two evaluations in order to answer the following research questions:

• RQ1: What is the effectiveness of CBT in generating valid test cases for real-world APIs with inter-parameter dependencies?
• RQ2: What is the fault-finding capability of CBT in real-world APIs with inter-parameter dependencies?

In this section, we detail the data collected when executing the experiments, what they consist of and where they are located.

Data location

All resources are placed inside the target directory. The <EVALUATION_NAME> identifies the experiment in terms of the service tested and how the test cases were generated. For example, youtubeSearch_allNominalIgnoreDependencies means that the YouTube-Search service was tested with 100% nominal test cases generated with RT, i.e., ignoring inter-parameter dependencies. See sections RQ1 and RQ2 for more information on the semantics of the identifiers used.

For each experiment performed, the following resources are provided:

• Allure reports. Location: target/allure-reports/<EVALUATION_NAME>/index.html. Graphical interface to visualize all test cases run on the API, which of them failed, and why. NOTE: Allure HTML files cannot be rendered locally, you must use a server for such purpose. Advice: open the files from an IDE such as IntelliJ IDEA.
• Test cases. Location: target/test-data/<EVALUATION_NAME>/test-cases.csv. These are a representation of the HTTP requests sent to the API, without the response.
• Test results. Location: target/test-data/<EVALUATION_NAME>/test-results.csv. These are a representation of the HTTP responses sent back by the API.
• Number of nominal and faulty test cases generated. Location: target/test-data/<EVALUATION_NAME>/nominal-faulty.csv.
• Time report. Location: target/test-data/<EVALUATION_NAME>/time.json. Includes information on the time taken for the whole experiment, the generation and the execution of test cases.
• Test coverage. Location: target/coverage-data/<EVALUATION_NAME>/test-coverage.json. Test coverage achieved by the test cases, according to the criteria defined in Test Coverage Criteria for RESTful Web APIs.

RQ1

For each service under test, we compared the effectiveness of random testing (RT) and constraint-based testing (CBT) to generate valid test cases. Interestingly, we found that some of the services tested had dependencies not described in the documentation. This was observed when obtaining 400 status codes ("bad request") with some input combinations that should be valid according to the documentation. In order to assess the effect of the missing dependencies, we defined them in the specification and included them in the evaluation as variants of the original case studies. Therefore, we performed the following three experiments for each service under test:

• allNominalIgnoreDependencies (RT). 100% of nominal test cases, i.e., those aiming to obtain a successful response (2XX status code), are generated. Inter-parameter dependencies are ignored.
• allNominalOriginalDependencies (CBT). 100% of nominal test cases are generated. Inter-parameter dependencies are taken into account for the generation of the test cases. IMPORTANT: The dependencies are specified as they are described in the documentation.
• allNominalCorrectDependencies (CBT). This experiment is equivalent to the previous one, but was necessary for those APIs whose dependencies are incorrectly specified in the documentation. IMPORTANT: As the dependencies are wrongly specified, they are fixed in order to generate the maximum number of valid test cases as possible. Inter-parameter dependencies are taken into account for the generation of the test cases.

RQ2

For each service under test, we compared the effectiveness of random testing (RT) and constraint-based testing (CBT) to find faults. We generated 1000 nominal test cases and 1000 faulty test cases, 2000 test cases in total. Faulty test cases in RT were generated by violating the specification of individual parameters, e.g., omitting a mandatory parameter in the API request. In CBT, however, faulty test cases were divided into two groups: 500 test cases following the same approach as in RT, and 500 test cases violating one or more inter-parameter dependencies. In total, we performed the following four experiments for each service under test:

• allNominalIgnoreDependencies (RT). Same experiment performed in RQ1.
• allFaultyParametersIgnoreDependencies (RT). 100% of faulty test cases, i.e., those aiming to obtain an error response (4XX status code) are generated. To create a faulty test case, two steps are performed: 1) generate a random request (ignoring dependencies); and 2) apply a random mutation to a single parameter so that the test case turns faulty, e.g., by removing a required parameter or passing a parameter the wrong type.
• allNominalOriginalDependencies (CBT). Same experiment performed in RQ1.
• allFaultyHalfHalfOriginalDependencies (CBT). 100% of faulty test cases are generated. These are divided into two groups: half of them are generated as in the previous evaluation; and the other half are generated by violating one or more inter-parameter dependencies. IMPORTANT: The dependencies are specified as they are described in the documentation.

4. Bugs found

Here we detail the bugs found in each service.

Foursquare - Search venues

• 500 status code, non-reproducible. A faulty request obtained a server error. Test ID: test_rh8b1qswv9di_searchVenues.
• Error response after valid request, reproducible. Multiple valid requests obtained the error Must provide parameters (ll and radius) or (sw and ne) or (near and radius) even when those conditions met. Test ID: test_1h7o4yd4erol5_searchVenues.
• Error response after valid request, reproducible. Multiple valid requests obtained the error Must provide parameter query even though all dependencies were satisfied and the parameter query was not involved in any of them. Test ID: test_1hrrikxnk2tlt_searchVenues.
• Successful response after request violating dependency, reproducible. Numerous requests violating dependencies obtain successful responses. In fact, we had to modify all dependencies present in this service, as they were incorrecly described in the documentation. Test ID: test_1h7fwrcprxbhd_searchVenues.
• Successful response after request with faulty parameter, reproducible. Requests using a radius higher than the maximum allowed (100000) obtain successful responses. Test ID: test_1h85apg914zea_searchVenues.
• Successful response after request with faulty parameter, reproducible. Requests using a limit higher than the maximum allowed (50) obtain successful responses. Test ID: test_1hr5dv8umqskx_searchVenues.
• Successful response after request with faulty parameter, reproducible. Requests using disallowed values for the parameter intent (enum) obtain successful responses. Test ID: test_1hrj8qnymvub7_searchVenues.
• OAS validation failure, reproducible. When removing the required parameter v, the status code 410 is returned, but such status code is not documented in the API specification. Test ID: test_1h7gdic67ko4w_searchVenues.
• OAS validation failure, reproducible. Some results returned by the API include properties not defined in the API specification, namely: response.confident, response.geocode, response.venues[].hasPerk", response.venues[].referralId, response.venues[].location.neighborhood. Test ID: test_1ib8pxoxamhx3_searchVenues.

GitHub - Get user repos

• Successful response after request with faulty parameter, reproducible. Any request that satisfies inter-parameter dependencies but uses invalid values for any parameter (e.g., an integer value for the parameter sort) obtains a successful response. This seems to be due to the fact that the service ignores any parameter using an invalid value. Test ID: test_1ib99sznvaxbr_getUserRepos.

Stripe - Create coupon

• Error response after valid request, reproducible. The Stripe documentation states that at least one of the parameters amount_off or percent_off must be used. However, those parameters are actually mutually exclusive. This causes requests that are supposedly valid to be rejected. Test ID: test_1h7geaqmn6lo5_PostCoupons.

Stripe - Create product

• Error response after valid request, reproducible. There exist two unspecified dependencies in this service. They cause one of every two supposedly valid requests to be rejected. Test ID: test_1h7fvyyo3bo12_PostProducts.

Tumblr - Get blog likes

• Successful response after request violating dependency, reproducible. Numerous requests violating dependencies obtain successful responses. This is because the dependencies of the service are actually 'softer' than what is described in the documentation, i.e., more parameters combinations are allowed. Test ID: test_1h7geaq9xlvtu_getBlogLikes.
• Successful response after request with faulty parameter, reproducible. Similarly to GitHub, any request that satisfies inter-parameter dependencies but uses invalid values for any parameter (e.g., a string value for the parameter before) obtains a successful response. This seems to be due to the fact that the service ignores any parameter using an invalid value. Test ID: test_1h7qzgkp2asz8_getBlogLikes.
• OAS validation failure, reproducible. Some results returned by the API include properties not defined in the API specification, namely: response._links, response.liked_posts[].blog, response.liked_posts[].id_string. Test ID: test_1h7fu18iwnmeo_getBlogLikes.

Yelp - Search bussinesses

• 500 status code, reproducible. The parameter open_at accepts a timestamp representing a future date. When sending a value higher than the maximum int32 value such as 2147483653, the API returns a server error. Test ID: test_1hrgf37q8oca8_getBusinesses.
• 500 status code, non-reproducible. Multiple requests (including valid and faulty ones) obtained server errors. Test ID: test_1hbd8xv6txw6w_getBusinesses.
• Error response after valid request, reproducible. When setting the location parameter to Egypt and the language to fi_FI (Finnish), the error LOCATION_NOT_FOUND is returned. However, changing the language makes the error disappear and actual results are returned. Test ID: test_r1cuuplspdnq_getBusinesses.
• Successful response after request violating dependency, reproducible. Requests using the parameters open_now and open_at set to false obtain successful responses, even though the API documentation explicitly states that both parameters cannot be used together. Test ID: test_1h7wh51d7ps1i_getBusinesses.
• OAS validation failure, reproducible. For some results returned by the API, the enum property businesses[].price includes values not enumerated in the documentation ([$, $$, $$$, $$$$]), for example, € and ￥￥￥. Test ID: test_trz3ptplfcj6_getBusinesses.
• OAS validation failure, reproducible. One result returned by the API has its businesses[].rating propoerty set to 0.0, even though the documentation explicitly states that this value must be between 1 and 5. Test ID: test_1iusr05s43p0l_getBusinesses.

YouTube - Get comment threads

• Error response after valid request, non-reproducible. A valid request obtained a client error categorised as "processingFailure", with the following message: The API server failed to successfully process the request. While this can be a transient error, it usually indicates that the requests input is invalid. Check the structure of the <code>commentThread</code> resource in the request body to ensure that it is valid.. Such message makes no sense, since this is a read operation and no body was included in the request. Test ID: test_qhiay3nt7t4j_youtubecommentThreadslist.
• Successful response after request violating dependency, reproducible. Requests using the parameters id and order obtain successful responses, even though the API documentation explicitly states that the order parameter is not supported for use in conjunction with the id parameter. Test ID: test_1h7j4wavuba79_youtubecommentThreadslist.
• OAS validation failure, reproducible. For some results returned by the API, the enum property items[].snippet.topLevelComment.snippet.moderationStatus is set to likelyspam, which is a value not enumerated in the documentation ([heldForReview, likelySpam, published, rejected]).

YouTube - Get videos

• Successful response after request violating dependency, reproducible. Requests using the parameter regionCode without also setting chart obtain successful responses, even though the API documentation explicitly states that the regionCode parameter can only be used in conjunction with the chart parameter. Test ID: test_1h7iolpche2ch_youtubevideoslist.
• Successful response after request violating dependency, reproducible. Requests using the parameters maxResults and id obtain successful responses, even though the API documentation explicitly states that the maxResults parameter is not supported for use in conjunction with the id parameter. Test ID: test_1h821gldox7w0_youtubevideoslist.
• Successful response after request violating dependency, reproducible. Requests using the parameter videoCategoryId without also setting chart obtain successful responses, even though the API documentation explicitly states that the videoCategoryId parameter can only be used in conjunction with the chart parameter. Test ID: test_1hbgh2v8tubed_youtubevideoslist.
• Successful response after request with faulty parameter, non-reproducible. In two occassions, requests ommitting the mandatory parameter part obtained successful responses. Test ID: test_s0xb0oqbliyg_youtubevideoslist.

YouTube - Search

• Error response after valid request, reproducible. There exist two unspecified dependencies in this service. They cause one of every two supposedly valid requests to be rejected. Test ID: test_1h7fu0woe00tv_youtubesearchlist.
• OAS validation failure, reproducible. Some results returned by the API include the property items[].snippet.publishTime, which is not defined in the API specification. Test ID: test_1hbghsasm15cz_youtubesearchlist.