Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add outbound-wasi-http-0.2.0 tests #2

Merged
merged 4 commits into from
Jun 14, 2024
Merged

Add outbound-wasi-http-0.2.0 tests #2

merged 4 commits into from
Jun 14, 2024

Conversation

rylev
Copy link
Collaborator

@rylev rylev commented Jun 13, 2024

This adds two tests that test outbound-http (with permissions and without).

This required a few new features to conformance testing:

  • Tests can now specify services they rely on.
  • Headers can contain templates in them that can be resolved against a test environment.
  • The conformance tests can be packaged into a directory without turning them into a tarball (good for local testing)

Here's a branch updating Spin to use the changes here: https://github.com/fermyon/spin/tree/update-conformance-tests

Signed-off-by: Ryan Levick <ryan.levick@fermyon.com>
Signed-off-by: Ryan Levick <ryan.levick@fermyon.com>
Comment on lines +16 to +19
{
"name": "transfer-encoding",
"value": "chunked"
},
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lann I was a bit surprised by this as the guest is just returning a buffer of bytes. I assume something funny is happening in wasi-http's implementation to cause this.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is expected when no Content-Length is given.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was able to confirm that indeed, if a Content-Length header is give, the transfer-encoding header disappears.

@rylev rylev force-pushed the package branch 2 times, most recently from a5f616a to 4c7840b Compare June 13, 2024 15:51
Signed-off-by: Ryan Levick <ryan.levick@fermyon.com>
crates/conformance-tests/src/config.rs Outdated Show resolved Hide resolved
Comment on lines +97 to +98
'outer: loop {
'inner: for captures in regex.captures_iter(content) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be less confusing if you break out the iterator, e.g.

Suggested change
'outer: loop {
'inner: for captures in regex.captures_iter(content) {
'outer: loop {
let mut captures_iter = regex.captures_iter(content);
'inner: for captures in captures_iter {

src/main.rs Show resolved Hide resolved
std::fs::copy(path, test_archive.join(&component_file))?;
if template_key.trim() == "source" {
manifest.replace_range(full.range(), &component_file);
'outer: loop {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the same code as replace_template above?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, I could certainly make the test-environment a dependency of this crate.

},
{
"name": "Date",
"optional": true
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this imply that this header check is exhaustive? I wonder if that's necessary or if we'd be better off just checking for certain headers (perhaps with e.g. null meaning "reject if this header exists").

🤷

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the header check is exhaustive. I'd like to stick with exhaustive checks for now, but I understand that as we try to document what runtimes are and are not allowed to do we might have to be less restrictive in the future.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel quite confident that runtimes won't return all the same headers, but waiting is fine.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I think the transfer-encoding example above is a good one here; it would have been semantically the same to return with no transfer-encoding (under certain circumstances...).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#3

Signed-off-by: Ryan Levick <ryan.levick@fermyon.com>
"path": "/",
"headers": [
{
"name": "url",

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason why we want to set the url in the header rather than as a query parameter or in the body?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope - is there any reason to prefer one over the other?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not particularly. I am not used to data being consumed by the service being in the header but to your point, I don't think it matters

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think headers are appropriate here; they are the most "metadatay" option in HTTP.

crates/conformance-tests/src/lib.rs Show resolved Hide resolved
.get(template_value)
.with_context(|| format!("'{template_value}' is not a known component"))?;
let component_file = "component.wasm";
std::fs::copy(path, test_archive.join(component_file))?;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this only evaluating the source under component.test? I think we need to update the request-shape app's spin.toml. Right now, i believe that the source file is not being replaced and the shim tests are failing with:

thread 'main' panicked at conformance-tests/src/main.rs:38:10:
called `Result::unwrap()` on an `Err` value: failed to copy file '/tmp/conformance-tests/request-shape/component.wasm' to temporary directory

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests work for me in the shim. Are you updating the shim conformance test binary to use the last version of the conformance-tests library?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was, but I'll give it another go

@rylev
Copy link
Collaborator Author

rylev commented Jun 14, 2024

Going to merge this as I don't want to hold up other PRs.

@rylev rylev merged commit 54a9615 into main Jun 14, 2024
2 checks passed
@rylev rylev deleted the package branch June 14, 2024 12:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants