Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

String truncation error while selecting from MON$ tables if some user-defined context variable exceeds 255 bytes in length [CORE5246] #5525

Closed
firebird-issue-importer opened this issue May 20, 2016 · 27 comments

Comments

@firebird-issue-importer

Submitted by: Volker Rehn (vr2_s18)

Trying to monitor statements by selecting from mon$statements. Start a recursive stored procedure, which runs all kinds of statements internally, execute statements, other SPs.
As long as the SP is not committed, any select from any monitoring table fails with sth like

Arithmetic overflow or division by zero has occurred.arithmetic exception, numeric overflow, or string truncation.
string right truncation.
expected length 255, actual 910.

The "actual" value varies between 300-something and 900-something, but is always higher than 255.

Seems like some place inside the monitoring tables logic can only handle 255 bytes but gets more.

Commits: e529c6a 373c459

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented May 23, 2016

Modified by: Volker Rehn (vr2_s18)

Component: Engine [ 10000 ]

environment: Firebird 3.0.0.32483 32 Bit, Win7 => Firebird 3.0.0.32483 32 Bit, Win7 64 Bit

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented May 23, 2016

Commented by: @dyemanov

Can you confirm that the problem happens when you select from any MON$ table, not necessarily MON$STATEMENTS?

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented May 24, 2016

Commented by: Volker Rehn (vr2_s18)

All monitoring tables are affected. Updated the description.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented May 24, 2016

Modified by: Volker Rehn (vr2_s18)

description: Trying to monitor statements by selecting from mon$statements. An application (web application, Flamerobin, doesn't matter) starts a stored procedure, which runs all kinds of statements internally.
Immediately after that I start a

select 1 from mon$statements

(this is of course reduced, but any other more realistic query fails as well)

I get - reliably -

Arithmetic overflow or division by zero has occurred.arithmetic exception, numeric overflow, or string truncation.
string right truncation.
expected length 255, actual 910.

The "actual" value varies between 600-something and 900-something, but is always higher than 255.

Seems like some place inside the mon$statements logic can only handle 255 bytes but gets more.

=>

Trying to monitor statements by selecting from mon$statements. Start a recursive stored procedure, which runs all kinds of statements internally, execute statements, other SPs.
As long as the SP is not committed, any select from any monitoring table fails with sth like

Arithmetic overflow or division by zero has occurred.arithmetic exception, numeric overflow, or string truncation.
string right truncation.
expected length 255, actual 910.

The "actual" value varies between 600-something and 900-something, but is always higher than 255.

Seems like some place inside the mon$statements logic can only handle 255 bytes but gets more.

summary: string right truncation with select 1 from mon$statements => string right truncation with monitoring tables

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented May 24, 2016

Modified by: Volker Rehn (vr2_s18)

description: Trying to monitor statements by selecting from mon$statements. Start a recursive stored procedure, which runs all kinds of statements internally, execute statements, other SPs.
As long as the SP is not committed, any select from any monitoring table fails with sth like

Arithmetic overflow or division by zero has occurred.arithmetic exception, numeric overflow, or string truncation.
string right truncation.
expected length 255, actual 910.

The "actual" value varies between 600-something and 900-something, but is always higher than 255.

Seems like some place inside the mon$statements logic can only handle 255 bytes but gets more.

=>

Trying to monitor statements by selecting from mon$statements. Start a recursive stored procedure, which runs all kinds of statements internally, execute statements, other SPs.
As long as the SP is not committed, any select from any monitoring table fails with sth like

Arithmetic overflow or division by zero has occurred.arithmetic exception, numeric overflow, or string truncation.
string right truncation.
expected length 255, actual 910.

The "actual" value varies between 300-something and 900-something, but is always higher than 255.

Seems like some place inside the mon$statements logic can only handle 255 bytes but gets more.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented May 24, 2016

Modified by: Volker Rehn (vr2_s18)

description: Trying to monitor statements by selecting from mon$statements. Start a recursive stored procedure, which runs all kinds of statements internally, execute statements, other SPs.
As long as the SP is not committed, any select from any monitoring table fails with sth like

Arithmetic overflow or division by zero has occurred.arithmetic exception, numeric overflow, or string truncation.
string right truncation.
expected length 255, actual 910.

The "actual" value varies between 300-something and 900-something, but is always higher than 255.

Seems like some place inside the mon$statements logic can only handle 255 bytes but gets more.

=>

Trying to monitor statements by selecting from mon$statements. Start a recursive stored procedure, which runs all kinds of statements internally, execute statements, other SPs.
As long as the SP is not committed, any select from any monitoring table fails with sth like

Arithmetic overflow or division by zero has occurred.arithmetic exception, numeric overflow, or string truncation.
string right truncation.
expected length 255, actual 910.

The "actual" value varies between 300-something and 900-something, but is always higher than 255.

Seems like some place inside the monitoring tables logic can only handle 255 bytes but gets more.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented May 24, 2016

Commented by: @dyemanov

The most probable source of problem is either remote client's host name or OS user name (non-ASCII and longish).

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented May 24, 2016

Commented by: Sean Leyne (seanleyne)

According to Windows MSDN "Computer Names" (https://msdn.microsoft.com/en-us/library/windows/desktop/ms724220(v=vs.85).aspx):

"DNS names consist of one or more components separated by a period (for example, http://msdn.microsoft.com). Each component can be up to 63 bytes. Each name can be up to 255 bytes total. DNS names are represented in the UTF-8 character set or Unicode."

Perhaps it is the UTF-8/Unicode encoding that is not being accounted for.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented May 24, 2016

Commented by: Volker Rehn (vr2_s18)

No, host or os user name are short and ascii. Plus I *can* monitor the SP in question if its nesting depth is minimal.

I narrowed it down further:

1. The error only happens with a recursive stored procedure
2. and only if the nesting depth is big enough, still small, already with maybe 5-10.

Something seems to accumulate in that case. I noticed that the "actual value" reported by the error msg goes up when I increase the nesting depth.
Also, within that SP, there are commands which rdb$set_context or rdb$get_context variables.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented May 24, 2016

Commented by: @dyemanov

MON$CONTEXT_VARIABLES expects context variables to hold strings up to 255 bytes, so this may also be a reason (e.g. nested calls concatenate something inside some context variable). RDB$GET_CONTEXT is prepared as returning VARCHAR(255). However, I don't see this limit enforced in RDB$SET_CONTEXT. So if the actual string in fact exceeds 255 bytes, MON$ tables will fail. I'm wondering whether such a limit was enforced in FB 2.5 or not. If not, FB 2.5 should raise the same error in this case.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented May 24, 2016

Modified by: @dyemanov

assignee: Dmitry Yemanov [ dimitr ]

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented May 24, 2016

Commented by: Volker Rehn (vr2_s18)

> So if the actual string in fact exceeds 255 bytes, MON$ tables will fail.

That's it. It is possible to set and get 2KB data without a problem.

> I'm wondering whether such a limit was enforced in FB 2.5 or not. If not, FB 2.5 should raise the same error in this case.

Or rather lift the limit on MON$CONTEXT_VARIABLES? Context vars provide a nice lightweight mechanism for transient data.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented May 25, 2016

Commented by: @dyemanov

Firebird 2.5 did not allow RDB$SET_CONTEXT to store variables longer than 255 bytes, it was a pseudo-UDF with the third parameter declared as VARCHAR(255), thus implicitly enforcing the limit. This is why MON$CONTEXT_VARIABLES also defines the variable value as VARCHAR(255). Firebird 3.0 has both RDB$ context functions reimplemented as native built-in functions, without enforcing any limits on the value length. So now context variables can be longer and MON$CONTEXT_VARIABLES is not prepared for that.

I can think of three possible solutions:

1) Restore the variable length limit of 255 bytes in v3.0. Applications that already utilized longer strings will be broken.
2) Increase the storage for MON$VARIABLE_VALUE. What should be the new limit? 1KB? 8KB? 32KB?
3) Truncate the actual variable value inside MON$VARIABLE_VALUE to 255 bytes.

(2) and (3) could also be combined, e.g. 1KB limit + truncation. Opinions?

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented May 25, 2016

Commented by: @pavel-zotov

> 2) Increase the storage for MON$VARIABLE_VALUE. What should be the new limit? 1KB? 8KB? 32KB?

Increase this field up to 65535 bytes, as we already have such limit for tables with regular varchar fields.

0xFF. And also I hope that at one nice day:
1) limit of 1024 context variables per session or per tx will gone.
2) one might to create ontext variables which store numbers, dates, booleans etc - i.e. we can get proper value without CAST().

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented May 25, 2016

Commented by: @dyemanov

1) Strings are limited to 32KB, not 64KB.
2) You probably won't be able to select from MON$ using older fbclient, due to 64KB message size restriction.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented May 25, 2016

Commented by: @asfernandes

4) Change MON$VARIABLE_VALUE to blob?

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented May 26, 2016

Commented by: Volker Rehn (vr2_s18)

Since Firebird 3 context functions don't enforce limits, blob would be consistent and minimize the need to check.

And truncation (at least with smaller limits) can easily turn into a source of bugs which are hard to find.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented May 26, 2016

Commented by: @dyemanov

The limit still exists, it's 32KB. I'd prefer to avoid blobs unless really necessary, as they are slow.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented May 26, 2016

Commented by: Volker Rehn (vr2_s18)

32K is fine, too - this is about consistency now.
Within my test case it was easy to let a TX GTT do the task of the overstretched context var.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jun 7, 2016

Modified by: @dyemanov

summary: string right truncation with monitoring tables => String truncation error while selecting from MON$ tables if some user-defined context variable exceeds 255 bytes in length

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jul 9, 2016

Modified by: @dyemanov

Version: 4.0 Initial [ 10621 ]

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jul 9, 2016

Commented by: @dyemanov

Please test the next snapshot build and report back.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jul 9, 2016

Modified by: @dyemanov

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Fixed [ 1 ]

Fix Version: 3.0.1 [ 10730 ]

Fix Version: 4.0 Alpha 1 [ 10731 ]

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jul 27, 2016

Modified by: @pavel-zotov

status: Resolved [ 5 ] => Resolved [ 5 ]

QA Status: No test => Done successfully

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jul 27, 2016

Modified by: @pavel-zotov

status: Resolved [ 5 ] => Closed [ 6 ]

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Apr 24, 2020

Modified by: @pavel-zotov

status: Closed [ 6 ] => Closed [ 6 ]

QA Status: Done successfully => Done with caveats

Test Details: RDB$GET_CONTEXT() truncates now values with length more than 255 chars.
In some cases this lead to regression - see letter from dimitr 24.04.2020 16:14.
Waiting for final solution of this problem.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jan 30, 2021

Modified by: @pavel-zotov

status: Closed [ 6 ] => Closed [ 6 ]

QA Status: Done with caveats => Done successfully

Test Details: RDB$GET_CONTEXT() truncates now values with length more than 255 chars.
In some cases this lead to regression - see letter from dimitr 24.04.2020 16:14.
Waiting for final solution of this problem.

=>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment