Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle bigint as string to prevent precision loss #353

Merged
merged 2 commits into from Jun 19, 2013

Conversation

sevastos
Copy link
Contributor

TIL dealing with bigints in javascript is a bit of a pain.

I had a weird issue of inserting correctly bigints on the db but getting off by a bit numbers on the response.

Apparently all the numbers in javascript are floating-point numbers1 and the accuracy is only assured for integers ±253 between -9007199254740992 and 9007199254740992.
However bigint ranges ±263: -9223372036854775808 to 9223372036854775807 2.

The most common solution for handling bigint data on javascript is carry them with strings and use special libs for math operations. So strings it is.

However, I wonder if it may cause any troubles to anyone that is treating bigint as number on javascript-level and do some checks around that. Though they shouldn't since that only works till the ±253 range and is not the right way.

Anyway let me know what you think.
Thanks

@rpedela
Copy link
Contributor

rpedela commented May 20, 2013

Related to #339.

@brianc
Copy link
Owner

brianc commented May 20, 2013

thanks for bringing this up. There's been discussion around it but nothings' really completely addressed it yet. I'll keep this open & really soon get to dealing with this more properly. Sorry for the headache you ran into so far!

},{
name: 'binary-bigint/int8-full',
format: 'binary',
dataTypeID: 20,
actual: [1, 0, 0, 0, 0, 0, 0, 102],
expected: 72057594037928030
expected: '72057594037928038'
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this wasn't correct.

1* 2567 + 102 = 72057594037928038

@sevastos
Copy link
Contributor Author

Thanks for letting me know @rpedela, for some reason, it didn't came up to my search before.

Hey @brianc, no worries and thanks for your work on this, great stuff. I am completely with you on having a consistent strategy on data type handling, I mostly did it for my project because it's quite urgent.

@brianc
Copy link
Owner

brianc commented May 23, 2013

Yeah so the only thing I can see is this requires a major version bump from 1.x to 2.x because it's a breaking change. There are a few other numeric problems execelently described by @rpedela in #339 which probably should be addressed before doing this major version bump since they're also backwards incompatible. I'll try to get all of them put in at the same time and merged. I'm not sure I'll have time for the next 10 days or so because I'm on vacation, but I'll try. If I don't get to it by then I'll try to get to it ASAP. If you were interested in appending the other number parsing changes to this pull request along with test modifications I could most likely merge it as soon as it was ready, but I understand if your time is limited & you can't get to the changes or dont want to. 😄 I'll eventually get to fixing this, it just might be 2 weeks or so.

Now, I'm off to the beach! 🏄

@sevastos
Copy link
Contributor Author

Sure, I'll give it a try, unless @rpedela wants to take it over. Enjoy your vacations 😃 !

Hey @rpedela, I haven't dealt with decimal and numeric but from what I see decimal is just an alias to numeric, you know if that's the case? Also the serial equivalents of the integer-variants are effectively the integers with a simple constraint for the autoincrement, so no need for new handlers.

@rpedela
Copy link
Contributor

rpedela commented May 23, 2013

Yes, decimal is an alias for numeric. And yes serial is just a column constraint. I added them for completeness relative to the Postgres number type docs.

I have been working non-stop the last couple months on a product launch set for the beginning of June. I will have more time starting mid-June. This is why I haven't done it already. Sorry about that. But if you want to do it then great!

@brianc
Copy link
Owner

brianc commented May 26, 2013

@rpedela no worries at all. If I get time on this vacation I'll knock it out - otherwise it'll be when I get back. Since you can override the type parsing with relative ease it isn't a critical thing but I do appreciate and share your concern over the correctness of the out-of-box behavior.

Also, good luck with your product launch!

@sevastos
Copy link
Contributor Author

Oops, I got interrupted.

So, the text parsers are pretty much done, but the binary parsers have some issues.

  • parseFloat32 suffers from loss of precision (e.x 101.1 => 101.0999984741211) but that's seems to be expected (according to IEEE754)
  • Parsing variabled-width numerics from binary responses looks quite the task to me

Unfortunately, I don't have time nor much knowledge on these to go more in depth, sorry about that.

@@ -82,7 +99,7 @@ var types = [{
// ignore some tests in binary mode
if (helper.config.binary) {
types = types.filter(function(type) {
return !(type.name in {'real':1, 'timetz':1, 'time':1, 'numeric': 1, 'double precision': 1});
return !(type.name in {'real': 1, 'timetz':1, 'time':1, 'numeric': 1});
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure about these.

parseFloat32 (real) somtimes has loss of precision. I tried Buffer's native function readFloat1 but the same precision loss happened.

However a code snippet from dzone :P returns correct amounts but it's probably slower and I haven't tested it thoroughly.

After reading some bits about floating points and IEEE754, it seems that sometimes loss of precision is expected.

@brianc
Copy link
Owner

brianc commented Jun 7, 2013

Hey @sevastos sorry for taking a long time to respond - I've been out.

First thing, don't worry about the binary parser. That was 100% community supplied by a couple of people & if they want to change their implementation they're more than welcome, but I don't support it myself. So, don't consider that a hold up by any means.

So, is anything outstanding other than the binary stuff? You can always skip the tests that are failing in binary mode. I've done it before. Whenever you're ready for me to review this without the binary stuff just let me know.

@sevastos
Copy link
Contributor Author

sevastos commented Jun 7, 2013

No problemo @brianc. Yeap, you can go ahead and review.

Cheers

@brianc
Copy link
Owner

brianc commented Jun 7, 2013

Okay - I reviewed this.

Just so I'm not missing anything, this is what I see as the changed datatypes

summary of changes

1. float4 -> parsed to javascript number (reverts change in v1.x, breaking change)
2. float8 -> parsed to javascript number (reverts change in v1.x, breaking change)
3. int2/4/oid -> parsed to javascript number (no change)
4. int8 -> parsed to javascript string (breaking change)
5. numeric -> parsed to javascript string (breaking change)
6. decimal -> parsed to javascript string (breaking change)

concerns

My only concern is including the ref dependency for the binary parser. I do appreciate the binary parser is correct, but like I said before I didn't write any of it and don't support it myself. Not a huge deal, but if ref not compiling starts giving windows users errors or something, I'll likely rip the binary stuff out completely into another module. For the mean time we can leave it in.

thoughts

I really appreciate everyone's help on this issue and the careful thought put into correct type handling. I apologize for not doing a better job of it myself: my own use of numeric types in applications I have done has never hit any of these pain points. Thankfully the open source & node community is full of friendly helpful folks to get things like this figured out for the greater good! 🎸

@rpedela since you were the originator of the well thought out issue #339 would you mind chiming in on this as well?
@booo any thoughts?

Once I hear from @rpedela I'll go ahead & do a major version bump, documentation update, and do what I can to broadcast this change out on twitter & IRC.

@rpedela
Copy link
Contributor

rpedela commented Jun 7, 2013

Yeah I think it looks great! I don't have any comment on the binary parser dependency.

Sorry I could not get to this sooner. On the bright side, my product was just released and it uses node-postgres.
https://www.datalanche.com/

@brianc
Copy link
Owner

brianc commented Jun 7, 2013

awesome! Congrats! No worries on taking a while. I'll get this merged this weekend & do the whole release song & dance.

@defunctzombie
Copy link
Contributor

Here is a small gist I created for wrapping numbers to avoid users accidentally "adding" them.

https://gist.github.com/shtylman/5483285

Not sure if this is a great solution but it is something to think about when working with numbers in js. I think we just need to be more vocal about the number problems for folks with more "accounting" sensitive applications.

@brianc
Copy link
Owner

brianc commented Jun 10, 2013

That's great @shtylman - I'll put big red letters somewhere about node-postgres returning a string if it's a big number and tell them to watch out and remind them to optionally drop in your exception throwing goodness.

@mariusa
Copy link

mariusa commented Mar 2, 2017

Would it be possible to please add a preference/setting, so that all bigint are parsed with parseInt(), knowing there will be data loss on really big numbers?

We constantly find bugs because in some place we forget data is transformed to string.

Thanks!

@eric-brechemier
Copy link

Apparently all the numbers in javascript are floating-point numbers1 and the accuracy is only assured for integers ±253 between -9007199254740992 and 9007199254740992.
However bigint ranges ±263: -9223372036854775808 to 9223372036854775807 2.

The most common solution for handling bigint data on javascript is carry them with strings and use special libs for math operations. So strings it is.

Why not keep BIGINT values as numbers up to ±253, and only use strings for values between ±253 and ±263?

This would be less confusing in most cases, and only confusing when truly required.

@rpedela
Copy link
Contributor

rpedela commented Mar 2, 2017

@mariusa Personally, I am not opposed to a config option. In the meantime, you can override the default type parser.

var pg = require('pg');

// bigint
pg.types.setTypeParser(20, function (value) {
    return parseInt(value);
});

// numeric
pg.types.setTypeParser(1700, function (value) {
    return parseFloat(value);
});

@mariusa
Copy link

mariusa commented Mar 2, 2017

Thanks, that works!

We'll trust the magic "20" number :) (saw the internal mapping, hope that won't change)

@rpedela
Copy link
Contributor

rpedela commented Mar 2, 2017

That magic number comes from SELECT oid FROM pg_type WHERE typname = 'int8'; and is unlikely to change.

Cruikshanks added a commit to DEFRA/sroc-charging-module-api that referenced this pull request Dec 10, 2020
https://trello.com/c/RIATlpAS

In the [charging-module-api](https://github.com/defra/charging-module-api) we hit an issue caused by integer overflow. We hold all our monetary values as pennies rather than pounds and pence. This is common practise in financial systems as it avoids some issues with rounding in equations.

The issue was all our monetary fields were set as `integer` but we found examples of bill runs with values larger than 2147483647 (the max an integer can hold). To solve the issue we needed to set the DB fields to `bigint`. [Bigint's](https://www.postgresql.org/docs/10/datatype-numeric.html#DATATYPE-INT) can hold values in the range of -9223372036854775808 to +9223372036854775807 in PostgreSQL.

JavaScript on the otherhand uses [Number](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number) to manage integers. An integer in JavaScript has a range of -9007199254740991 to 9007199254740991. If you need to go as large as PostgreSQL you need to use a [JavaScript BigInt](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/BigInt).

Because of this discrepency the PostgreSQL driver for Node by default returns all BigInt values as strings. This is a [design decision by PostgreSQL](brianc/node-postgres#353) to avoid any loss of precision.

However, if you are confident you won't be effected by this precision loss (which we are) you can tell the PostgreSQL driver to return an integer instead.

So as we do not expect to generate bill runs with values greater than 9.0071993e+13 (I don't even know how to say what that number is!) this change adds the config needed so we can forget about BigInt's in the rest of our code.
Cruikshanks added a commit to DEFRA/sroc-charging-module-api that referenced this pull request Dec 10, 2020
https://trello.com/c/RIATlpAS

In the [charging-module-api](https://github.com/defra/charging-module-api) we hit an issue caused by integer overflow. We hold all our monetary values as pennies rather than pounds and pence. This is common practise in financial systems as it avoids some issues with rounding in equations.

The issue was all our monetary fields were set as `integer` but we found examples of bill runs with values larger than 2147483647 (the max an integer can hold). To solve the issue we needed to set the DB fields to `bigint`. [Bigint's](https://www.postgresql.org/docs/10/datatype-numeric.html#DATATYPE-INT) can hold values in the range of -9223372036854775808 to +9223372036854775807 in PostgreSQL.

JavaScript on the other hand uses [Number](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number) to manage integers. An integer in JavaScript has a range of -9007199254740991 to 9007199254740991. If you need to go as large as PostgreSQL you need to use a [JavaScript BigInt](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/BigInt).

Because of this discrepancy, the PostgreSQL driver for Node by default returns all BigInt values as strings. This is a [design decision by PostgreSQL](brianc/node-postgres#353) to avoid any loss of precision.

However, if you are confident you won't be affected by this precision loss (which we are) you can tell the PostgreSQL driver to return an integer instead.

So as we do not expect to generate bill runs with values greater than 9.0071993e+13 (I don't even know how to say what that number is!) this change adds the config needed so we can forget about BigInt's in the rest of our code.
fuxingloh pushed a commit to JellyfishSDK/jellyfish that referenced this pull request Aug 25, 2022
[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [pg](https://togithub.com/brianc/node-postgres) | [`8.7.3` -> `8.8.0`](https://renovatebot.com/diffs/npm/pg/8.7.3/8.8.0) | [![age](https://badges.renovateapi.com/packages/npm/pg/8.8.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/npm/pg/8.8.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/npm/pg/8.8.0/compatibility-slim/8.7.3)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/npm/pg/8.8.0/confidence-slim/8.7.3)](https://docs.renovatebot.com/merge-confidence/) |

---

### Release Notes

<details>
<summary>brianc/node-postgres</summary>

### [`v8.8.0`](https://togithub.com/brianc/node-postgres/blob/HEAD/CHANGELOG.md#pg880)

[Compare Source](https://togithub.com/brianc/node-postgres/compare/pg@8.7.3...pg@8.8.0)

-   Bump minimum required version of [native bindings](https://togithub.com/brianc/node-postgres/pull/2787)
-   Catch previously uncatchable errors thrown in [`pool.query`](https://togithub.com/brianc/node-postgres/pull/2569)
-   Prevent the pool from blocking the event loop if all clients are [idle](https://togithub.com/brianc/node-postgres/pull/2721) (and `allowExitOnIdle` is enabled)
-   Support `lock_timeout` in [client config](https://togithub.com/brianc/node-postgres/pull/2779)
-   Fix errors thrown in callbacks from [interfering with cleanup](https://togithub.com/brianc/node-postgres/pull/2753)

##### pg-pool@3.5.0

-   Add connection [lifetime limit](https://togithub.com/brianc/node-postgres/pull/2698) config option.

##### pg@8.7.0

-   Add optional config to [pool](https://togithub.com/brianc/node-postgres/pull/2568) to allow process to exit if pool is idle.

##### pg-cursor@2.7.0

-   Convert to [es6 class](https://togithub.com/brianc/node-postgres/pull/2553)
-   Add support for promises [to cursor methods](https://togithub.com/brianc/node-postgres/pull/2554)

##### pg@8.6.0

-   Better [SASL](https://togithub.com/brianc/node-postgres/pull/2436) error messages & more validation on bad configuration.
-   Export [DatabaseError](https://togithub.com/brianc/node-postgres/pull/2445).
-   Add [ParameterDescription](https://togithub.com/brianc/node-postgres/pull/2464) support to protocol parsing.
-   Fix typescript [typedefs](https://togithub.com/brianc/node-postgres/pull/2490) with `--isolatedModules`.

##### pg-query-stream@4.0.0

-   Library has been [converted](https://togithub.com/brianc/node-postgres/pull/2376) to Typescript. The behavior is identical, but there could be subtle breaking changes due to class names changing or other small inconsistencies introduced by the conversion.

##### pg@8.5.0

-   Fix bug forwarding [ssl key](https://togithub.com/brianc/node-postgres/pull/2394).
-   Convert pg-query-stream internals to [typescript](https://togithub.com/brianc/node-postgres/pull/2376).
-   Performance [improvements](https://togithub.com/brianc/node-postgres/pull/2286).

##### pg@8.4.0

-   Switch to optional peer dependencies & remove [semver](https://togithub.com/brianc/node-postgres/commit/a02dfac5ad2e2abf0dc3a9817f953938acdc19b1) package which has been a small thorn in the side of a few users.
-   Export `DatabaseError` from [pg-protocol](https://togithub.com/brianc/node-postgres/commit/58258430d52ee446721cc3e6611e26f8bcaa67f5).
-   Add support for `sslmode` in the [connection string](https://togithub.com/brianc/node-postgres/commit/6be3b9022f83efc721596cc41165afaa07bfceb0).

##### pg@8.3.0

-   Support passing a [string of command line options flags](https://togithub.com/brianc/node-postgres/pull/2216) via the `{ options: string }` field on client/pool config.

##### pg@8.2.0

-   Switch internal protocol parser & serializer to [pg-protocol](https://togithub.com/brianc/node-postgres/tree/master/packages/pg-protocol). The change is backwards compatible but results in a significant performance improvement across the board, with some queries as much as 50% faster. This is the first work to land in an on-going performance improvment initiative I'm working on. Stay tuned as things are set to get much faster still! 🚀

##### pg-cursor@2.2.0

-   Switch internal protocol parser & serializer to [pg-protocol](https://togithub.com/brianc/node-postgres/tree/master/packages/pg-protocol). The change is backwards compatible but results in a significant performance improvement across the board, with some queries as much as 50% faster.

##### pg-query-stream@3.1.0

-   Switch internal protocol parser & serializer to [pg-protocol](https://togithub.com/brianc/node-postgres/tree/master/packages/pg-protocol). The change is backwards compatible but results in a significant performance improvement across the board, with some queries as much as 50% faster.

##### pg@8.1.0

-   Switch to using [monorepo](https://togithub.com/brianc/node-postgres/tree/master/packages/pg-connection-string) version of `pg-connection-string`. This includes better support for SSL argument parsing from connection strings and ensures continuity of support.
-   Add `&ssl=no-verify` option to connection string and `PGSSLMODE=no-verify` environment variable support for the pure JS driver. This is equivalent of passing `{ ssl: { rejectUnauthorized: false } }` to the client/pool constructor. The advantage of having support in connection strings and environment variables is it can be "externally" configured via environment variables and CLI arguments much more easily, and should remove the need to directly edit any application code for [the SSL default changes in 8.0](https://node-postgres.com/announcements#&#8203;2020-02-25). This should make using `pg@8.x` significantly less difficult on environments like Heroku for example.

##### pg-pool@3.2.0

-   Same changes to `pg` impact `pg-pool` as they both use the same connection parameter and connection string parsing code for configuring SSL.

##### pg-pool@3.1.0

-   Add [maxUses](https://togithub.com/brianc/node-postgres/pull/2157) config option.

##### pg@8.0.0

##### note: for detailed release notes please [check here](https://node-postgres.com/announcements#&#8203;2020-02-25)

-   Remove versions of node older than `6 lts` from the test matrix. `pg>=8.0` may still work on older versions but it is no longer officially supported.
-   Change default behavior when not specifying `rejectUnauthorized` with the SSL connection parameters. Previously we defaulted to `rejectUnauthorized: false` when it was not specifically included. We now default to `rejectUnauthorized: true.` Manually specify `{ ssl: { rejectUnauthorized: false } }` for old behavior.
-   Change [default database](https://togithub.com/brianc/node-postgres/pull/1679) when not specified to use the `user` config option if available. Previously `process.env.USER` was used.
-   Change `pg.Pool` and `pg.Query` to [be](https://togithub.com/brianc/node-postgres/pull/2126) an [es6 class](https://togithub.com/brianc/node-postgres/pull/2063).
-   Make `pg.native` non enumerable.
-   `notice` messages are [no longer instances](https://togithub.com/brianc/node-postgres/pull/2090) of `Error`.
-   Passwords no longer [show up](https://togithub.com/brianc/node-postgres/pull/2070) when instances of clients or pools are logged.

##### pg@7.18.0

-   This will likely be the last minor release before pg@8.0.
-   This version contains a few bug fixes and adds a deprecation warning for [a pending change in 8.0](https://togithub.com/brianc/node-postgres/issues/2009#issuecomment-579371651) which will flip the default behavior over SSL from `rejectUnauthorized` from `false` to `true` making things more secure in the general use case.

##### pg-query-stream@3.0.0

-   [Rewrote stream internals](https://togithub.com/brianc/node-postgres/pull/2051) to better conform to node stream semantics. This should make pg-query-stream much better at respecting [highWaterMark](https://nodejs.org/api/stream.html#stream_new_stream_readable_options) and getting rid of some edge case bugs when using pg-query-stream as an async iterator. Due to the size and nature of this change (effectively a full re-write) it's safest to bump the semver major here, though almost all tests remain untouched and still passing, which brings us to a breaking change to the API....
-   Changed `stream.close` to `stream.destroy` which is the [official](https://nodejs.org/api/stream.html#stream_readable_destroy_error) way to terminate a readable stream. This is a **breaking change** if you rely on the `stream.close` method on pg-query-stream...though should be just a find/replace type operation to upgrade as the semantics remain very similar (not exactly the same, since internals are rewritten, but more in line with how streams are "supposed" to behave).
-   Unified the `config.batchSize` and `config.highWaterMark` to both do the same thing: control how many rows are buffered in memory. The `ReadableStream` will manage exactly how many rows are requested from the cursor at a time. This should give better out of the box performance and help with efficient async iteration.

##### pg@7.17.0

-   Add support for `idle_in_transaction_session_timeout` [option](https://togithub.com/brianc/node-postgres/pull/2049).

##### 7.16.0

-   Add optional, opt-in behavior to test new, [faster query pipeline](https://togithub.com/brianc/node-postgres/pull/2044). This is experimental, and not documented yet. The pipeline changes will grow significantly after the 8.0 release.

##### 7.15.0

-   Change repository structure to support lerna & future monorepo [development](https://togithub.com/brianc/node-postgres/pull/2014).
-   [Warn about deprecation](https://togithub.com/brianc/node-postgres/pull/2021) for calling constructors without `new`.

##### 7.14.0

-   Reverts 7.13.0 as it contained [an accidental breaking change](https://togithub.com/brianc/node-postgres/pull/2010) for self-signed SSL cert verification. 7.14.0 is identical to 7.12.1.

##### 7.13.0

-   Add support for [all tls.connect()](https://togithub.com/brianc/node-postgres/pull/1996) options.

##### 7.12.0

-   Add support for [async password lookup](https://togithub.com/brianc/node-postgres/pull/1926).

##### 7.11.0

-   Add support for [connection_timeout](https://togithub.com/brianc/node-postgres/pull/1847/files#diff-5391bde944956870128be1136e7bc176R63) and [keepalives_idle](https://togithub.com/brianc/node-postgres/pull/1847).

##### 7.10.0

-   Add support for [per-query types](https://togithub.com/brianc/node-postgres/pull/1825).

##### 7.9.0

-   Add support for [sasl/scram authentication](https://togithub.com/brianc/node-postgres/pull/1835).

##### 7.8.0

-   Add support for passing [secureOptions](https://togithub.com/brianc/node-postgres/pull/1804) SSL config.
-   Upgrade [pg-types](https://togithub.com/brianc/node-postgres/pull/1806) to 2.0.

##### 7.7.0

-   Add support for configurable [query timeout](https://togithub.com/brianc/node-postgres/pull/1760) on a client level.

##### 7.6.0

-   Add support for ["bring your own promise"](https://togithub.com/brianc/node-postgres/pull/1518)

##### 7.5.0

-   Better [error message](https://togithub.com/brianc/node-postgres/commit/11a4793452d618c53e019416cc886ad38deb1aa7) when passing `null` or `undefined` to `client.query`.
-   Better [error handling](https://togithub.com/brianc/node-postgres/pull/1503) on queued queries.

##### 7.4.0

-   Add support for [Uint8Array](https://togithub.com/brianc/node-postgres/pull/1448) values.

##### 7.3.0

-   Add support for [statement timeout](https://togithub.com/brianc/node-postgres/pull/1436).

##### 7.2.0

-   Pinned pg-pool and pg-types to a tighter semver range. This is likely not a noticeable change for you unless you were specifically installing older versions of those libraries for some reason, but making it a minor bump here just in case it could cause any confusion.

##### 7.1.0

##### Enhancements

-   [You can now supply both a connection string and additional config options to clients.](https://togithub.com/brianc/node-postgres/pull/1363)

##### 7.0.0

##### Breaking Changes

-   Drop support for node < `4.x`.
-   Remove `pg.connect` `pg.end` and `pg.cancel` singleton methods.
-   `Client#connect(callback)` now returns `undefined`. It used to return an event emitter.
-   Upgrade [pg-pool](https://togithub.com/brianc/node-pg-pool) to `2.x`.
-   Upgrade [pg-native](https://togithub.com/brianc/node-pg-native) to `2.x`.
-   Standardize error message fields between JS and native driver. The only breaking changes were in the native driver as its field names were brought into alignment with the existing JS driver field names.
-   Result from multi-statement text queries such as `SELECT 1; SELECT 2;` are now returned as an array of results instead of a single result with 1 array containing rows from both queries.

[Please see here for a migration guide](https://node-postgres.com/guides/upgrading)

##### Enhancements

-   Overhauled documentation: <https://node-postgres.com>.
-   Add `Client#connect() => Promise<void>` and `Client#end() => Promise<void>` calls. Promises are now returned from all async methods on clients *if and only if* no callback was supplied to the method.
-   Add `connectionTimeoutMillis` to pg-pool.

##### v6.2.0

-   Add support for [parsing `replicationStart` messages](https://togithub.com/brianc/node-postgres/pull/1271/files).

##### v6.1.0

-   Add optional callback parameter to the pure JavaScript `client.end` method. The native client already supported this.

##### v6.0.0

##### Breaking Changes

-   Remove `pg.pools`. There is still a reference kept to the pools created & tracked by `pg.connect` but it has been renamed, is considered private, and should not be used. Accessing this API directly was uncommon and was *supposed* to be private but was incorrectly documented on the wiki. Therefore, it is a breaking change of an (unintentionally) public interface to remove it by renaming it & making it private. Eventually `pg.connect` itself will be deprecated in favor of instantiating pools directly via `new pg.Pool()` so this property should become completely moot at some point. In the mean time...check out the new features...

##### New features

-   Replace internal pooling code with [pg-pool](https://togithub.com/brianc/node-pg-pool). This is the first step in eventually deprecating and removing the singleton `pg.connect`. The pg-pool constructor is exported from node-postgres at `require('pg').Pool`. It provides a backwards compatible interface with `pg.connect` as well as a promise based interface & additional niceties.

You can now create an instance of a pool and don't have to rely on the `pg` singleton for anything:

    var pg = require('pg')

    var pool = new pg.Pool()

    // your friendly neighborhood pool interface, without the singleton
    pool.connect(function(err, client, done) {
      // ...
    })

Promise support & other goodness lives now in [pg-pool](https://togithub.com/brianc/node-pg-pool).

**Please** read the readme at [pg-pool](https://togithub.com/brianc/node-pg-pool) for the full api.

-   Included support for tcp keep alive. Enable it as follows:

```js
var client = new Client({ keepAlive: true })
```

This should help with backends incorrectly considering idle clients to be dead and prematurely disconnecting them.

##### v5.1.0

-   Make the query object returned from `client.query` implement the promise interface. This is the first step towards promisifying more of the node-postgres api.

Example:

```js
var client = new Client()
client.connect()
client.query('SELECT $1::text as name', ['brianc']).then(function (res) {
  console.log('hello from', res.rows[0])
  client.end()
})
```

##### v5.0.0

##### Breaking Changes

-   `require('pg').native` now returns null if the native bindings cannot be found; previously, this threw an exception.

##### New Features

-   better error message when passing `undefined` as a query parameter
-   support for `defaults.connectionString`
-   support for `returnToHead` being passed to [generic pool](https://togithub.com/coopernurse/node-pool)

##### v4.5.0

-   Add option to parse JS date objects in query parameters as [UTC](https://togithub.com/brianc/node-postgres/pull/943)

##### v4.4.0

-   Warn to `stderr` if a named query exceeds 63 characters which is the max length supported by postgres.

##### v4.3.0

-   Unpin `pg-types` semver. Allow it to float against `pg-types@1.x`.

##### v4.2.0

-   Support for additional error fields in postgres >= 9.3 if available.

##### v4.1.0

-   Allow type parser overrides on a [per-client basis](https://togithub.com/brianc/node-postgres/pull/679)

##### v4.0.0

-   Make [native bindings](https://togithub.com/brianc/node-pg-native.git) an optional install with `npm install pg-native`
-   No longer surround query result callback with `try/catch` block.
-   Remove built in COPY IN / COPY OUT support - better implementations provided by [pg-copy-streams](https://togithub.com/brianc/node-pg-copy-streams.git) and [pg-native](https://togithub.com/brianc/node-pg-native.git)

##### v3.6.0

-   Include support for (parsing JSONB)\[brianc/node-pg-types#13] (supported in postgres 9.4)

##### v3.5.0

-   Include support for parsing boolean arrays

##### v3.4.0

-   Include port as connection parameter to [unix sockets](https://togithub.com/brianc/node-postgres/pull/604)
-   Better support for odd [date parsing](https://togithub.com/brianc/node-pg-types/pull/8)

##### v3.2.0

-   Add support for parsing [date arrays](https://togithub.com/brianc/node-pg-types/pull/3)
-   Expose array parsers on [pg.types](https://togithub.com/brianc/node-pg-types/pull/2)
-   Allow [pool](https://togithub.com/brianc/node-postgres/pull/591) to be configured

##### v3.1.0

-   Add [count of the number of times a client has been checked out from the pool](https://togithub.com/brianc/node-postgres/pull/556)
-   Emit `end` from `pg` object [when a pool is drained](https://togithub.com/brianc/node-postgres/pull/571)

##### v3.0.0

##### Breaking changes

-   [Parse the DATE PostgreSQL type as local time](https://togithub.com/brianc/node-postgres/pull/514)

After [some discussion](https://togithub.com/brianc/node-postgres/issues/510) it was decided node-postgres was non-compliant in how it was handling DATE results. They were being converted to UTC, but the PostgreSQL documentation specifies they should be returned in the client timezone. This is a breaking change, and if you use the `date` type you might want to examine your code and make sure nothing is impacted.

-   [Fix possible numeric precision loss on numeric & int8 arrays](https://togithub.com/brianc/node-postgres/pull/501)

pg@v2.0 included changes to not convert large integers into their JavaScript number representation because of possibility for numeric precision loss. The same types in arrays were not taken into account. This fix applies the same type of type-coercion rules to arrays of those types, so there will be no more possible numeric loss on an array of very large int8s for example. This is a breaking change because now a return type from a query of `int8[]` will contain *string* representations
of the integers. Use your favorite JavaScript bignum module to represent them without precision loss, or punch over the type converter to return the old style arrays again.

-   [Fix to input array of dates being improperly converted to utc](https://togithub.com/benesch/node-postgres/commit/c41eedc3e01e5527a3d5c242fa1896f02ef0b261#diff-7172adb1fec2457a2700ed29008a8e0aR108)

Single `date` parameters were properly sent to the PostgreSQL server properly in local time, but an input array of dates was being changed into utc dates. This is a violation of what PostgreSQL expects. Small breaking change, but none-the-less something you should check out if you are inserting an array of dates.

-   [Query no longer emits `end` event if it ends due to an error](https://togithub.com/brianc/node-postgres/commit/357b64d70431ec5ca721eb45a63b082c18e6ffa3)

This is a small change to bring the semantics of query more in line with other EventEmitters. The tests all passed after this change, but I suppose it could still be a breaking change in certain use cases. If you are doing clever things with the `end` and `error` events of a query object you might want to check to make sure its still behaving normally, though it is most likely not an issue.

##### New features

-   [Supercharge `prepareValue`](https://togithub.com/brianc/node-postgres/pull/555)

The long & short of it is now any object you supply in the list of query values will be inspected for a `.toPostgres` method. If the method is present it will be called and its result used as the raw text value sent to PostgreSQL for that value. This allows the same type of custom type coercion on query parameters as was previously afforded to query result values.

-   [Domain aware connection pool](https://togithub.com/brianc/node-postgres/pull/531)

If domains are active node-postgres will honor them and do everything it can to ensure all callbacks are properly fired in the active domain. If you have tried to use domains with node-postgres (or many other modules which pool long lived event emitters) you may have run into an issue where the active domain changes before and after a callback. This has been a longstanding footgun within node-postgres and I am happy to get it fixed.

-   [Disconnected clients now removed from pool](https://togithub.com/brianc/node-postgres/pull/543)

Avoids a scenario where your pool could fill up with disconnected & unusable clients.

-   [Break type parsing code into separate module](https://togithub.com/brianc/node-postgres/pull/541)

To provide better documentation and a clearer explanation of how to override the query result parsing system we broke the type converters [into their own module](https://togithub.com/brianc/node-pg-types). There is still work around removing the 'global-ness' of the type converters so each query or connection can return types differently, but this is a good first step and allow a lot more obvious way to return int8 results as JavaScript numbers, for example

##### v2.11.0

-   Add support for [application_name](https://togithub.com/brianc/node-postgres/pull/497)

##### v2.10.0

-   Add support for [the password file](http://www.postgresql.org/docs/9.3/static/libpq-pgpass.html)

##### v2.9.0

-   Add better support for [unix domain socket](https://togithub.com/brianc/node-postgres/pull/487) connections

##### v2.8.0

-   Add support for parsing JSON\[] and UUID\[] result types

##### v2.7.0

-   Use single row mode in native bindings when available \[[@&#8203;rpedela](https://togithub.com/rpedela)]
    -   reduces memory consumption when handling row values in 'row' event
-   Automatically bind buffer type parameters as binary \[[@&#8203;eugeneware](https://togithub.com/eugeneware)]

##### v2.6.0

-   Respect PGSSLMODE environment variable

##### v2.5.0

-   Ability to opt-in to int8 parsing via `pg.defaults.parseInt8 = true`

##### v2.4.0

-   Use eval in the result set parser to increase performance

##### v2.3.0

-   Remove built-in support for binary Int64 parsing.
    *Due to the low usage & required compiled dependency this will be pushed into a 3rd party add-on*

##### v2.2.0

-   [Add support for excapeLiteral and escapeIdentifier in both JavaScript and the native bindings](https://togithub.com/brianc/node-postgres/pull/396)

##### v2.1.0

-   Add support for SSL connections in JavaScript driver
-   this means you can connect to heroku postgres from your local machine without the native bindings!
-   [Add field metadata to result object](https://togithub.com/brianc/node-postgres/blob/master/test/integration/client/row-description-on-results-tests.js)
-   [Add ability for rows to be returned as arrays instead of objects](https://togithub.com/brianc/node-postgres/blob/master/test/integration/client/results-as-array-tests.js)

##### v2.0.0

-   Properly handle various PostgreSQL to JavaScript type conversions to avoid data loss:

<!---->

    PostgreSQL | pg@v2.0 JavaScript | pg@v1.0 JavaScript
    --------------------------------|----------------
    float4     | number (float)     | string
    float8     | number (float)     | string
    int8       | string             | number (int)
    numeric    | string             | number (float)
    decimal    | string             | number (float)

For more information see [brianc/node-postgres#353
If you are unhappy with these changes you can always [override the built in type parsing fairly easily](https://togithub.com/brianc/node-pg-parse-float).

##### v1.3.0

-   Make client_encoding configurable and optional

##### v1.2.0

-   return field metadata on result object: access via result.fields\[i].name/dataTypeID

##### v1.1.0

-   built in support for `JSON` data type for PostgreSQL Server @&#8203; v9.2.0 or greater

##### v1.0.0

-   remove deprecated functionality
    -   Callback function passed to `pg.connect` now **requires** 3 arguments
    -   Client#pauseDrain() / Client#resumeDrain removed
    -   numeric, decimal, and float data types no longer parsed into float before being returned. Will be returned from query results as `String`

##### v0.15.0

-   client now emits `end` when disconnected from back-end server
-   if client is disconnected in the middle of a query, query receives an error

##### v0.14.0

-   add deprecation warnings in prep for v1.0
-   fix read/write failures in native module under node v0.9.x

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

 **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox.

---

This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/JellyfishSDK/jellyfish).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzMi4xNzMuMCIsInVwZGF0ZWRJblZlciI6IjMyLjE3My4wIn0=-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants