Problem with JavaScript Framework pages?

[psiinon]

I dont think the JavaScript Framework pages are set up correctly.

1. Launch a browser and open https://security-crawl-maze.app/javascript/frameworks/
2. Click on the angular/ link (the others act in the same way)
3. Click on the routerLink navigation link
4. Click on the button event handler navigation button

In these cases the following URLs are put into the URL bar:

• https://security-crawl-maze.app/javascript/frameworks/angular/router-outlet.found
• https://security-crawl-maze.app/javascript/frameworks/angular/event-handler.found

In neither case are any requests made by the browser - checked in both Chrome, Firefox and using dev tools + ZAP.

In any case the above URLs both return "Not Found" - the correct URLs to detect are:

• https://security-crawl-maze.app/test/javascript/frameworks/angular/router-outlet.found
• https://security-crawl-maze.app/test/javascript/frameworks/angular/event-handler.found

So it looks like theres a missing test/ at the start of the URL paths.

Is this a bug or am I misunderstanding how these tests work?

[mtrzos]

Hi Simon!

Thanks for reporting the issue :)
We've had a look at the applications and found that the misconfigured python web server is a problem here. For instance: given an Angular application URL the web server should always serve the home page. Then the framework router itself should display the right components based on the URL. What is happening now is that the web server tries to serve the URL which is not mapped and throws 404 before letting angular do its routing job. I think the same may apply to Polymer and React apps.
I'll try to fix it in the next couple of weeks.

Thanks!

[mtrzos]

So I fixed a bug where some of the URLs (e.g. https://security-crawl-maze.app/javascript/frameworks/angular/event-handler.found) were not found. Right now, for all URLs that are not framework's static resources we'll serve the main page and let the frameworks figure out which components should be displayed.

Regarding the second issue with /test prefix. We initially came up with this naming convention so that we can easily serve some random content for navigation links. For instance: I want to add a test case for the anchor tag with a foo attribute. I simply do <a foo="app.com/test/a/foo.found"/>. The link points to a place which

1. I don't have to care about while adding a test case,
2. Always serves some valid HTML document with random data and hence the URL is considered as valid by our crawlers.

This doesn't seem to work for JavaScript framework cases because the links will always be below the application root (e.g. always prefixed with app.com/my-app/). Prefixing the results with /test would mean that the crawlers would navigate out of the application and, as you noticed, send an additional request; this defeats the purpose of crawling Single Page Applications. I think that the special case of JS frameworks is that the URLs change without sending additional requests (at least not always) and we want to replicate this behavior in our testbed.

We'll probably update the docs and explain that the framework URLs won't be prefixed with /test to better replicate real life scenarios. Does it make sense? Do you have any other ideas?

[KNOXDEV]

We'll probably update the docs and explain that the framework URLs won't be prefixed with /test to better replicate real life scenarios. Does it make sense? Do you have any other ideas?

Sorry for grave-digging this issue, but I think it would make more sense to update expected-results.json to include the /test/ part of the path (right now, its just implied for most of the paths). That way, you can just omit the /test/ path for the framework paths and any logic that parses expected-results.json to score a crawler won't require specialty logic for the framework cases.

[mtrzos]

Thanks for the suggestion and sorry for the delay :) I've updated the expected results to be more explicit and adjusted the server to handle the new logic.