Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent crawler from changing url #85

Open
Pau1fitz opened this issue Sep 23, 2020 · 0 comments
Open

Prevent crawler from changing url #85

Pau1fitz opened this issue Sep 23, 2020 · 0 comments

Comments

@Pau1fitz
Copy link

Do you want to request a feature or report a bug?

What is the current behavior?

When using the crawler I have a url with an ! included the following url gets created:
https://www.test.com/liverpool/players%21/

however, I am looking for the url to look like the following:

https://www.test.com/liverpool/players!/

If the current behavior is a bug, please provide the steps to reproduce.

My code looks like the following:

const SitemapGenerator = require("sitemap-generator");

// create generator
const generator = SitemapGenerator("http://www.test.com", {
    stripQuerystring: false,
    decodeResponses: true
});

// register event listeners
generator.on("done", () => {
    // sitemaps created
    console.log("sitemaps created");
});

// start the crawler
generator.start();

When running node generator.js the url is being replaced with %21 for !

What is the expected behavior?

I want to keep the original url i.e. - https://www.test.com/liverpool/players!/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant