Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Force Crawling of TurboTax Free File Edition #5

wants to merge 1 commit into from
Changes from all commits
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.


Just for now

@@ -263,13 +263,20 @@ void EmitKeyValueToHandler(int line, const ParsedRobotsKey& key,
absl::string_view value,
RobotsParseHandler* handler) {
typedef ParsedRobotsKey Key;
switch (key.type()) {

// Stop TurboTax from hiding their free file edition
Key changedKey = key;
if (absl::StrContains(value, "taxfreedom")) {
changedKey = Key::ALLOW;

switch (changedKey.type()) {
case Key::USER_AGENT: handler->HandleUserAgent(line, value); break;
case Key::ALLOW: handler->HandleAllow(line, value); break;
case Key::DISALLOW: handler->HandleDisallow(line, value); break;
case Key::SITEMAP: handler->HandleSitemap(line, value); break;
case Key::UNKNOWN:
handler->HandleUnknownAction(line, key.GetUnknownText(), value);
handler->HandleUnknownAction(line, changedKey.GetUnknownText(), value);
// No default case Key:: to have the compiler warn about new values.
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.