Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Precision of FLOAT #2627

Open
randomJoe211 opened this issue Jul 20, 2021 · 0 comments
Open

Precision of FLOAT #2627

randomJoe211 opened this issue Jul 20, 2021 · 0 comments
Labels
type/enhancement Type: make the code neat or more efficient

Comments

@randomJoe211
Copy link
Contributor

Please check the FAQ documentation before raising an issue

Please check the FAQ documentation and old issues before raising an issue in case someone has asked the same question that you are asking.

Describe the bug (must be provided)

When I store a decimal as the value of a property, if the defined type of the property if FLOAT and the length of the decimal exceeds the precision of FLOAT, Nebula Graph seem to have altered it to a double and, more importantly, altered the last several digits randomly, instead of putting them into 0 or simply trimming the number.

nGQL:

(root@nebula) [dt_test]> insert vertex player (name, score, score1, score2, score3, score4, score5, score6) VALUE 1:("Joe", 0xb, 0xb, 0xb, 0xb, 0xb, 3.12345678901234567890, 5.1234567890123456789);
Execution succeeded (time spent 1502/1998 us)

Tue, 20 Jul 2021 08:51:20 UTC

(root@nebula) [dt_test]> FETCH PROP ON player 1

+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
| vertices_                                                                                                                                                    |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
| (1 :player{name: "Joe", score: 11, score1: 11, score2: 11, score3: 11, score4: 11, score5: 3.1234567165374756, score6: 5.123456789012345, score7: __NULL__}) |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
Got 1 rows (time spent 1663/2391 us)

Tue, 20 Jul 2021 08:51:56 UTC

(root@nebula) [dt_test]> show create tag player

+----------+-----------------------------------+
| Tag      | Create Tag                        |
+----------+-----------------------------------+
| "player" | "CREATE TAG `player` (            |
|          |  `name` fixed_string(32) NULL,    |
|          |  `score` int64 NULL,              |
|          |  `score1` int8 NULL,              |
|          |  `score2` int16 NULL,             |
|          |  `score3` int32 NULL,             |
|          |  `score4` int64 NULL,             |
|          |  `score5` float NULL,             |
|          |  `score6` double NULL,            |
|          |  `score7` bool NULL               |
|          | ) ttl_duration = 0, ttl_col = """ |
+----------+-----------------------------------+
Got 1 rows (time spent 1639/2461 us)

Note that score5 is now 3.1234567165374756, which is odd.

@CPWstatic CPWstatic transferred this issue from vesoft-inc/nebula-graph Aug 27, 2021
@CPWstatic CPWstatic added type/bug Type: something is unexpected type/enhancement Type: make the code neat or more efficient labels Aug 28, 2021
@Sophie-Xie Sophie-Xie removed the type/bug Type: something is unexpected label Aug 30, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement Type: make the code neat or more efficient
Projects
None yet
Development

No branches or pull requests

3 participants