http.agent.name is dropped from the robots.txt match set when http.robots.agents is set

[rzo1]

RobotRulesParser.setConf() never adds http.agent.name to agentNames when http.robots.agents is non-empty:

• if the advertised name is listed first, index++ skips it and the loop appends only agents[1..], so the name is dropped entirely;
• if it is not listed first, the code logs "Agent we advertise (X) not listed first" and appends the configured list, still without adding the name.

agentNames is only ever written in the empty-list branch and in that loop, which contradicts the in-code comment: "If both are present, our agent-string should be the first one we advertise to robots-parsing."

Impact

With http.agent.name: mybot and http.robots.agents: mybot,mybot-uk, the match set handed to crawler-commons is [mybot-uk]. A User-agent: mybot group in a robots.txt does not match and the parser falls back to *. Rules addressed specifically to the crawler are ignored, which is a compliance issue rather than a cosmetic one.

Reproducer

With a robots.txt of:

User-agent: mybot
Disallow: /restricted/

User-agent: *
Disallow: /

and http.agent.name: mybot, http.robots.agents: mybot,mybot-uk, /index.html is reported as disallowed because the wildcard group is selected.

Nothing in core/src/test currently references http.robots.agents.

Fix

Add agentNames.add(agentName) before the append loop. agentNames is a LinkedHashSet, so the existing index++ can stay and no duplicate is introduced.