Questions about "fn_varbintohexstr" (MSSQL, ASP)

First of all, I want to thank sqlmap project

When data is received from MSSQL + ASP using master.dbo.fn_varbintohexstr,
The web response is 404.

If query does NOT contain master.dbo.fn_varbintohexstr,
the response is 200 OK, but the query does not work.

Probably not a problem with SQLMAP or a firewall.
I want to know the cause of this problem and how to solve it.

I am sorry for my poor English. Thank you.

[stamparm]

You should try to find out what is going on. Run sqlmap with -v 3 and try to reproduce the "problem" inside the web browser with shown [PAYLOAD] queries. I really can't tell whether something is triggering the problem (e.g. master.dbo inside the WAF) or the DBMS has problems with that function.

A) You can try (not suggesting that THIS is the solution) to replace the <hex> tag (https://github.com/sqlmapproject/sqlmap/blob/master/xml/queries.xml#L162) inside the xml/queries.xml at your sqlmap installation with something like:

<hex query="sys.fn_varbintohexstr(CAST(%s AS VARBINARY(8000)))"/>

B) You can try to rerun your case with --no-escape

p.s. as said, without you trying the case manually I can only imagine what is going on