diff --git a/devrev-snapin-template.plain b/devrev-snapin-template.plain index 31d7100..e2e5f08 100644 --- a/devrev-snapin-template.plain +++ b/devrev-snapin-template.plain @@ -60,6 +60,7 @@ - The Conformance Tests should spin a server at http://localhost:8002 for testing callbacks (The Callback Server). - The Conformance Tests *MUST NOT* send the event directly to The Callback Server. + - The Implementation Code *MUST NOT* send the event directly to The Callback Server. - The DevRev Server runs at http://localhost:8003, so it must not be mocked. diff --git a/run_devrev_snapin_conformance_tests.sh b/run_devrev_snapin_conformance_tests.sh index f04d506..5491dfc 100755 --- a/run_devrev_snapin_conformance_tests.sh +++ b/run_devrev_snapin_conformance_tests.sh @@ -6,9 +6,9 @@ NPM_INSTALL_OUTPUT_FILTER="up to date in|added [0-9]* packages, removed [0-9]* p ANSI_ESCAPE_PATTERN="s/\x1b\[[0-9;]*[mK]//g" # Maximum number of characters to display from log files -SNAP_IN_LOG_MAX_CHARS=30000 -DEVREV_SERVER_LOG_MAX_CHARS=15000 -PROXY_SERVER_LOG_MAX_CHARS=10000 +SNAP_IN_LOG_MAX_CHARS=80000 +DEVREV_SERVER_LOG_MAX_CHARS=40000 +PROXY_SERVER_LOG_MAX_CHARS=20000 # Function to print a log file, truncating it if it's too large print_log_file() { diff --git a/templates/attachment_extraction_acceptance_test.plain b/templates/attachment_extraction_acceptance_test.plain index e6e91fe..156a185 100644 --- a/templates/attachment_extraction_acceptance_test.plain +++ b/templates/attachment_extraction_acceptance_test.plain @@ -1,10 +1,10 @@ - - Test The Extraction Function with the following flow (The Attachment Test Flow): - - Step 1: Invoke The Extraction Function using the resource [{{ data_extraction_resource_name }}](test_data/{{ data_extraction_resource_name }}). - - Expect The Callback Server to receive from DevRev a **single** event with "event_type" that equals "EXTRACTION_DATA_DONE". - - Step 2: Invoke The Extraction Function using the resource [{{ attachment_extraction_resource_name }}](test_data/{{ attachment_extraction_resource_name }}). - - Expect The Callback Server to receive from DevRev a **single** event (`:= event_from_callback_server`) with "event_type" that equals "EXTRACTION_ATTACHMENTS_DONE". - - Expect `event_from_callback_server["event_data"]["artifacts"]` to be an array. Expect this array to not be empty. Expect this array to have length 1. - - Let `artifact_object = event_from_callback_server["event_data"]["artifacts"][0]`. - - Expect `artifact_object["item_type"] = "ssor_attachment"` and `artifact_object["item_count"] = {{ expected_attachment_count }}`. - - Make a GET request to "http://localhost:8003/is_uploaded/{artifact_object['id']}". Expect response code to be equal to 200. - - Note: The Attachment Test Flow should be executed within a single test. \ No newline at end of file +Test The Extraction Function with the following flow (The Attachment Test Flow): + - Step 1: Invoke The Extraction Function using the resource [{{ data_extraction_resource_name }}](test_data/{{ data_extraction_resource_name }}). + - Expect The Callback Server to receive from DevRev a **single** event with "event_type" that equals "EXTRACTION_DATA_DONE". + - Step 2: Invoke The Extraction Function using the resource [{{ attachment_extraction_resource_name }}](test_data/{{ attachment_extraction_resource_name }}). + - Expect The Callback Server to receive from DevRev a **single** event (`:= event_from_callback_server`) with "event_type" that equals "EXTRACTION_ATTACHMENTS_DONE". + - Expect `event_from_callback_server["event_data"]["artifacts"]` to be an array. Expect this array to not be empty. Expect this array to have length 1. + - Let `artifact_object = event_from_callback_server["event_data"]["artifacts"][0]`. + - Expect `artifact_object["item_type"] = "ssor_attachment"` and `artifact_object["item_count"] = {{ expected_attachment_count }}`. + - Make a GET request to "http://localhost:8003/is_uploaded/{artifact_object['id']}". Expect response code to be equal to 200. + - Note: The Attachment Test Flow should be executed within a single test. \ No newline at end of file diff --git a/templates/chef_cli_normalization_validation.plain b/templates/chef_cli_normalization_validation.plain index 19a8cb8..101f428 100644 --- a/templates/chef_cli_normalization_validation.plain +++ b/templates/chef_cli_normalization_validation.plain @@ -1,10 +1,10 @@ - - Then, validate The Normalization Function with the chef-cli tool (The Chef CLI) using the command "chef-cli validate-data -m -r {{ record_type }}": - - To invoke the function that returns The External Domain Metadata JSON object, send the event [external_domain_metadata_event_payload.json](test_data/external_domain_metadata_event_payload.json) to The Snap-In server. - - The External Domain Metadata JSON object should then be stored in temporary file and provided as path to the validation of The Normalization Function. - - The Chef CLI reads the extracted file (The Extracted File) from stdin. - - The Extracted File is inside folder provided in the environment variable "EXTRACTED_FILES_FOLDER_PATH". To retrieve the relevant file name, command `ls EXTRACTED_FILES_FOLDER_PATH | grep extractor_{{ record_type }} | sort -r | head -n 1` should be executed. The test should always fail if The Extracted File is not present. - - Please note that the expected behaviour is that only after the invocation of The Extraction Function, the folder provided in the environment variable "EXTRACTED_FILES_FOLDER_PATH" should exist. The test should always fail if before the execution of The Chef CLI, the folder "EXTRACTED_FILES_FOLDER_PATH" does not exist or if The Extracted File is not found. - - The path to The Chef CLI executable is provided in the environment variable `CHEF_CLI_PATH`. - - For the validation of The Normalization Function to be successful The Chef CLI must return an empty output. The implementation of The Extraction Function directly infuences the output of The Chef CLI. - - The stdout and stderr of The Chef CLI *must* be printed to the console. - - The test should always fail if The Chef CLI is not available. \ No newline at end of file +validate The Normalization Function with the chef-cli tool (The Chef CLI) using the command "chef-cli validate-data -m -r {{ record_type }}": + - To invoke the function that returns The External Domain Metadata JSON object, send the event [external_domain_metadata_event_payload.json](test_data/external_domain_metadata_event_payload.json) to The Snap-In server. + - The External Domain Metadata JSON object should then be stored in temporary file and provided as path to the validation of The Normalization Function. + - The Chef CLI reads the extracted file (The Extracted File) from stdin. + - The Extracted File is inside folder provided in the environment variable "EXTRACTED_FILES_FOLDER_PATH". To retrieve the relevant file name, command `ls EXTRACTED_FILES_FOLDER_PATH | grep extractor_{{ record_type }} | sort -r | head -n 1` should be executed. The test should always fail if The Extracted File is not present. + - Please note that the expected behaviour is that only after the invocation of The Extraction Function, the folder provided in the environment variable "EXTRACTED_FILES_FOLDER_PATH" should exist. The test should always fail if before the execution of The Chef CLI, the folder "EXTRACTED_FILES_FOLDER_PATH" does not exist or if The Extracted File is not found. + - The path to The Chef CLI executable is provided in the environment variable `CHEF_CLI_PATH`. + - For the validation of The Normalization Function to be successful The Chef CLI must return an empty output. The implementation of The Extraction Function directly infuences the output of The Chef CLI. + - The stdout and stderr of The Chef CLI *must* be printed to the console. + - The test should always fail if The Chef CLI is not available. \ No newline at end of file diff --git a/templates/data_extraction_acceptance_test.plain b/templates/data_extraction_acceptance_test.plain index 012b89d..c517aca 100644 --- a/templates/data_extraction_acceptance_test.plain +++ b/templates/data_extraction_acceptance_test.plain @@ -1,6 +1,6 @@ - - Test The Extraction Function using the resource [{{ resource_name }}](test_data/{{ resource_name }}). - - Expect The Callback Server to receive from DevRev a **single** event with "event_type" that equals "EXTRACTION_DATA_DONE". - - Let `artifact_array = ["event_data"]["artifacts"]` (array of objects). - - Expect `len(artifact_array) > 0`. - - Out of `artifact_array`, expect one of the elements to have "item_type" equal to "{{ artifact_name }}" (`:= {{ artifact_name }}_artifact`) and `{{ artifact_name }}_artifact["item_count"]={{ expected_item_count }}`. - - Note: If `{{ artifact_name }}_artifact["item_count"] < {{ expected_item_count }}`, this indicates that not all the {{ artifact_name }} data was extracted. \ No newline at end of file +Test The Extraction Function using the resource [{{ resource_name }}](test_data/{{ resource_name }}). + - Expect The Callback Server to receive from DevRev a **single** event with "event_type" that equals "EXTRACTION_DATA_DONE". + - Let `artifact_array = ["event_data"]["artifacts"]` (array of objects). + - Expect `len(artifact_array) > 0`. + - Out of `artifact_array`, expect one of the elements to have "item_type" equal to "{{ artifact_name }}" (`:= {{ artifact_name }}_artifact`) and `{{ artifact_name }}_artifact["item_count"]={{ expected_item_count }}`. + - Note: If `{{ artifact_name }}_artifact["item_count"] < {{ expected_item_count }}`, this indicates that not all the {{ artifact_name }} data was extracted. \ No newline at end of file diff --git a/templates/test_rate_limiting_during_data_extraction.plain b/templates/test_rate_limiting_during_data_extraction.plain index ee183fd..64ed389 100644 --- a/templates/test_rate_limiting_during_data_extraction.plain +++ b/templates/test_rate_limiting_during_data_extraction.plain @@ -1,7 +1,7 @@ - - Test the rate limiting of The Function "{{ function_name }}" with the following flow: - - Make a POST request to "http://localhost:8004/start_rate_limiting" with body `{ "test_name": }`. - - Invoke The Function "{{ function_name }}" with valid credentials and all required parameters. - - Expect: `status_code = 429`. - - Expect: `api_delay > 0` and `api_delay <= 3`. If api_delay is bigger than 3, assume the problem is that we did not calculate the api_delay correctly in The Implementation Code. - - Make a POST request to "http://localhost:8004/end_rate_limiting". +Test the rate limiting of The Function "{{ function_name }}" with the following flow: + - Make a POST request to "http://localhost:8004/start_rate_limiting" with body `{ "test_name": }`. + - Invoke The Function "{{ function_name }}" with valid credentials and all required parameters. + - Expect: `status_code = 429`. + - Expect: `api_delay > 0` and `api_delay <= 3`. If api_delay is bigger than 3, assume the problem is that we did not calculate the api_delay correctly in The Implementation Code. + - Make a POST request to "http://localhost:8004/end_rate_limiting". \ No newline at end of file diff --git a/templates/test_rate_limiting_during_emitting_events.plain b/templates/test_rate_limiting_during_emitting_events.plain index c01ecb6..7744647 100644 --- a/templates/test_rate_limiting_during_emitting_events.plain +++ b/templates/test_rate_limiting_during_emitting_events.plain @@ -1,5 +1,5 @@ - - When the input event is "{{ input_event_name }}", the Extraction Function must handle rate limiting using the following test flow: - - Step 1: Make a POST request to "http://localhost:8004/start_rate_limiting" with body `{ "test_name": }`. - - Step 2: Invoke The Extraction Function using the resource [{{ resource_name }}](test_data/{{ resource_name }}). - - Expect The Callback Server to receive *a single* event with "event_type" "{{ expected_output_event_type }}". - - Step 3: Make a POST request to "http://localhost:8004/end_rate_limiting". \ No newline at end of file +When the input event is "{{ input_event_name }}", the Extraction Function must handle rate limiting using the following test flow: + - Step 1: Make a POST request to "http://localhost:8004/start_rate_limiting" with body `{ "test_name": }`. + - Step 2: Invoke The Extraction Function using the resource [{{ resource_name }}](test_data/{{ resource_name }}). + - Expect The Callback Server to receive *a single* event with "event_type" "{{ expected_output_event_type }}". + - Step 3: Make a POST request to "http://localhost:8004/end_rate_limiting". \ No newline at end of file