-
-
Notifications
You must be signed in to change notification settings - Fork 6.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Aurora Data API #4375
feat: Aurora Data API #4375
Conversation
Yes, we can add a new driver to TypeORM. But please be sure to keep to maintain it.
please try to resolve driver-specific requirements in the driver itself. Sometimes its possible, but just requires non standard solution. Maybe you can came-up with some solution in the driver? |
…so most of the queries should work, added some tests
I implemented a PoC code that transforms typeorm interface into data API internally, so most of the queries should work now. There's definitely some improvements to be done, but at least it works. @pleerock what's the best way to make functional tests work? As I understand currently they are ran on the machine with the database installed locally, but Aurora is only available in cloud so there should be an AWS account involved. |
Related with #4305 Let me know if I can contribute with something. I also need this driver and was thinking of doing myself and then saw yours :)) |
@Can-Sahin I think right now it should work for most of the queries. The only thing blocking the merge I believe is the ability to run tests on CI which requires action from the maintainers. Basically, we need an Aurora running in an AWS account and the credentials setup in the environment variables. I can provide a server but I think it would make more sense if it was controlled by one of the maintainers. I can of course help with the setup. We can also skip the functional tests for now and fix them in a separate PR. @pleerock @daniel-lang @Kononnable @rustamwin Thoughts? |
I think |
@Can-Sahin You're absolutely correct. I added the transaction support to the TODO list which we can later migrate to an issue or wherever we decide to keep the list. |
Will the data-api suport postgres too? |
@pontusab I have no info but I think AWS will implement it at some point. They released Serverless Postgres today. I would expect the interface to be the same for both so this should work. |
@ArsenyYankovsky Awesome! Looking forward to using this 💯 |
I actually think I might create a separate repo with some of the code like converting the queries to live there. It will simplify the life of TypeORM devs and will allow us to iterate with the fixes faster. It will still require some of the data API-related code to live in TypeORM though. |
I ve been following data api since first beta and its quite new. I suppose there will be changes over time and everytime it needs to be merged in typeorm repo and wait for release. Maybe a seperate package that deals with core parts make sense but still it will not get typeorm release cycle faster. Another reason for #2479 i guess |
I moved a lot of Aurora-specific code into a separate driver here. I also set the functional and unit tests up against a real database and CI/CD for that repo. Also I've started tracking the TODO list in the issues section of the driver repo. |
I implemented transaction support in the driver and will add some documentation and examples soon |
Hey all, I'm also super keen to get this into typeorm. Any idea if it could be merged? is there anything blocking it? Happy to help out however I can :) |
@pleerock could you please take another look at this PR? It's ready to merge from my side and there's plenty of people wishing to use it asap. Thanks. |
@pleerock It would be amazing to pull this in soon. Any thoughts on ETA? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good for me.
About tests: I hate to say it but lets skip tests for this driver (for now). Running tests against real AWS DB should tests slow down and also there could be some request per second limitations on AWS side.
Thanks for taking a look. I have tests running in the driver repo against real database and latest typeorm. |
Merged :) Totally a big heap for serverless sql development. Good goooood job @ArsenyYankovsky 🎉 |
Awesome! |
Greetings, is this PR available though npm? Or is this another way to test it in my project until it goes live? I tried installing both (btw, awesome job and thanks a lot for this implementation) |
Hi @Schavras, it isn't rolled into a new release yet, so you'll need to build both manually. Here's what I did:
You'll still need typeorm and the aurora driver in your package.json. You'll only need to do this for a little while until @pleerock creates either a tag or an official release. Until then, installing the direct commit hash won't give you access to the new connection type. Hope this helps! |
Hey, I have this working from AWS lambda to MySQL Aurora serverless via Data API. Amazing work! However I am frequently receiving: Has anyone else been experiencing this? This is of course a bit of a showstopper for me however if I can get this resolved everything else seems to be working perfectly. Any ideas? It is worth mentioning perhaps that I have keepConnectionAlive set as true as I was receiving connection already open errors. FWIW I am also using with nestjs. @pleerock would be really awesome to cut a release for this to save hacking about with builds. Thanks ever so much for all your efforts! Awesome stuff. |
@TreeMan360 For me this ususally happens if my database is paused. It takes some time for the aurora to start up again. I've created a question on aws forum here to make this error retryable somehow. |
Also eagerly waiting on this making it into the npm releases 😄 ! |
+1 on awaiting the npm release 😁 |
commit 7623e890609610c1ea8cc6f98f70556267d8a8c6 Merge: 038c2edd 13ac222 Author: Petter Häggholm <petter@petterhaggholm.net> Date: Sat Sep 14 17:54:03 2019 -0700 bugfix commit 038c2edd31def7860e46e8f3c92cfc88d7e66c61 Author: Petter Häggholm <petter@petterhaggholm.net> Date: Sat Sep 14 14:22:35 2019 -0700 fix locking commit 13ac222 Author: Umed Khudoiberdiev <pleerock.me@gmail.com> Date: Fri Sep 13 23:20:36 2019 +0500 updated changelog commit ae1f131 Author: Umed Khudoiberdiev <pleerock.me@gmail.com> Date: Fri Sep 13 21:22:40 2019 +0500 version bump commit d8f1c81 Author: Roman <52884896+roman-acumen@users.noreply.github.com> Date: Fri Sep 13 18:12:23 2019 +0300 fix: sqlite connections don't ignore the schema property (typeorm#4599) commit 0094f61 Author: Dennie de Lange <dennie@tkvw.nl> Date: Fri Sep 13 16:58:20 2019 +0200 feat: add referenced table metadata to NamingStrategy to resolve foreign key name (typeorm#4274) Feature allows to use foreignkey metadata to generate database foreignkey names. Closes typeorm#3847 and typeorm#1355 commit 3abe5b9 Author: Toby Hinloopen <toby@bonaroo.nl> Date: Fri Sep 13 16:04:12 2019 +0200 fix: "hstore injection" & properly handle NULL, empty string, backslashes & quotes in hstore key/value pairs (typeorm#4720) * Improve HStore object support * Add hstore-injection test commit 644c21b Author: Muma David Bwalya <davidtheprogrammer42@gmail.com> Date: Fri Sep 13 12:32:22 2019 +0200 docs: explicitly defining the optional nature of the ManyToOne and OneToMany relationship when viewed from the child entity (typeorm#4722) * Update many-to-one-one-to-many-relations.md Explicitly defining the optional nature of the ManyToOne and OneToMany relationship when viewed from the child entity. * Update docs/many-to-one-one-to-many-relations.md Punctuation and more accurate semantics. Co-Authored-By: Toby Hinloopen <toby@bonaroo.nl> commit c52b3d2 Author: Evgeniy <evgesh1918@gmail.com> Date: Fri Sep 13 13:04:11 2019 +0300 fix: views generating broken Migrations (typeorm#4726) MigrationGenerateCommand didn't apply parameters to query and information about ViewTables couldn't be inserted into typeorm_metadata table. Also added code that creates typeorm_metadata table if ViewTables exists Fixed issue typeorm#4123 commit 1d73a90 Author: Ian Mobley <ianmobley@gmail.com> Date: Fri Sep 13 00:22:17 2019 -0700 fix: createQueryBuilder relation remove works only if using ID (typeorm#2632) (typeorm#4734) commit 81f4b43 Author: Alex Howard <thezanke@gmail.com> Date: Fri Sep 13 03:16:07 2019 -0400 docs: added missing comma in relations.md (typeorm#4739) commit 7808bba Author: Junggun Lim <junggun.lim@gmail.com> Date: Thu Sep 5 12:44:25 2019 -0700 feat: UpdateResult returns affected rows in postgresql (typeorm#4432) * Added 'affected' field in UpdateResult as well as in DeleteResult. * PostgresQueryRunner returns the number of affected rows properly * UpdateQueryBuilder retrieves the affected rows returned by PostgresQueryRunner and sets the added 'affected' field of UpdateResult properly. Closes: typeorm#1308 commit 7a0beed Author: Mophy Xiong <mophy.xh@gmail.com> Date: Fri Sep 6 03:33:39 2019 +0800 fix: the excessive stack depth comparing types `FindConditions<?>` and `FindConditions<?>` problem (typeorm#4470) commit dacac83 Author: Michał Wadas <michalwadas@gmail.com> Date: Thu Sep 5 21:28:56 2019 +0200 feat: add materialized View support for Postgres (typeorm#4478) feat: add option to synchronize or not to synchronize ViewEntity Fixes typeorm#4317 Fixes typeorm#3996 commit db8074a Author: Nicolas Hervé <nicolas.herve.pessac@gmail.com> Date: Thu Sep 5 19:42:52 2019 +0200 feat: add support for ON CONFLICT for cockroach (typeorm#4518) Closes: typeorm#4513 commit 19e2179 Author: hauau <hauau@users.noreply.github.com> Date: Fri Sep 6 00:38:54 2019 +0700 feat: add `set` datatype support for MySQL/MariaDB (typeorm#4538) Set possible values defined using existing enum column option. Sets are implemented as arrays. Closes: typeorm#2779 commit 5c311ed Author: David Chen <42685381+DavidChen-minted@users.noreply.github.com> Date: Thu Sep 5 08:50:10 2019 -0700 feat: add options to input init config for sql.js (typeorm#4560) * add options to input init config for sql.js * update changelog * updated connection-options docs commit 3cf470d Author: Aviad Hadad <aviadhadad@gmail.com> Date: Thu Sep 5 18:46:19 2019 +0300 fix: change PrimaryColumn decorator to clone passed options (typeorm#4571) Closes: typeorm#4570 commit 9e3d664 Author: Mike Guida <mike@mguida.com> Date: Thu Sep 5 08:38:41 2019 -0700 docs: clarify title of configuration sources (typeorm#4592) commit 587d534 Author: Thomas Gieling <thomas.gieling@gmail.com> Date: Thu Sep 5 17:37:43 2019 +0200 use remove in stead of delete (typeorm#4574) the `remove` method requires the entity itself as input. To remove a record based on it's id, you need to use `delete` commit a925be9 Author: Michał Wadas <michalwadas@gmail.com> Date: Thu Sep 5 16:05:23 2019 +0200 feat: add postgres pool error handler (typeorm#4474) Add option to customize pool error handling. Users can decide to log these errors with higher level (eg. error), crash application or reconnect. commit f65ecc7 Author: JB Reefer <james.reefer@gmail.com> Date: Thu Sep 5 09:10:10 2019 -0400 docs: grammar and wording fixes in active-record-data-mapper.md (typeorm#4615) Grammar and wording fixes in bottom paragraph commit 445c740 Author: Duckie <7842848+duckies@users.noreply.github.com> Date: Thu Sep 5 08:59:50 2019 -0400 fix: apostrophe in Postgres enum strings breaks query (typeorm#4631) * Patch to allow apostrophes in postgres enum string. * Make linter happy * Testing fix for MySQL * Limit to postgres and mysql, fix test description * Lets not be greedy. commit c1406bb Author: kevindashgit <friedheim.kevin@gmail.com> Date: Thu Sep 5 05:30:24 2019 -0700 docs: documentation for showMigrations() (typeorm#4644) commit 2478198 Author: azxj <31400828+azxj@users.noreply.github.com> Date: Thu Sep 5 20:29:23 2019 +0800 docs: remove duplicate segment in entities.md (typeorm#4638) Remove duplicate segment. commit 5bd29d5 Author: QoVoQ <674263588@qq.com> Date: Thu Sep 5 20:27:30 2019 +0800 docs: update many-to-many-relations.md, make it easier to understand (typeorm#4680) * Update many-to-many-relations.md Make the example of the last section `many-to-many relations with custom properties` more clear and general. * Update many-to-many-relations.md commit 92e4270 Author: Coroliov Oleg <1880059+ruscon@users.noreply.github.com> Date: Thu Sep 5 15:25:06 2019 +0300 feat: add mongodb `useUnifiedTopology` config parameter (typeorm#4684) commit 690e6f5 Author: CavidM <cavid2409@gmail.com> Date: Thu Sep 5 05:18:03 2019 -0700 fix: "database" option error in driver when use "url" option for connection commit e589fda Author: Michael Dzjaparidze <michael.dzjaparidze@gmail.com> Date: Tue Sep 3 15:05:58 2019 +0200 feat: export additional schema builder classes (typeorm#4325) commit 3951b58 Author: David Podhola <david.podhola@naseukoly.cz> Date: Sat Aug 31 23:39:04 2019 +0200 docs: fix missing async (typeorm#4458) In the second example the lambda is correctly started with an `async`. It is missing in the first example. commit d4e2443 Author: Kateile <sylvanuskateile@gmail.com> Date: Sat Aug 31 21:26:44 2019 +0300 docs: update typeorm-model-shim.js comment link (typeorm#4540) This link http://webpack.github.io/docs/configuration.html#resolve no longer exist commit 4c2bffc Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri Aug 30 23:43:04 2019 +0200 chore: bump lodash from 4.17.11 to 4.17.15 (typeorm#4671) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.11 to 4.17.15. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.11...4.17.15) Signed-off-by: dependabot[bot] <support@github.com> commit 10bac1f Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri Aug 30 23:35:40 2019 +0200 build: bump mixin-deep from 1.3.1 to 1.3.2 (typeorm#4648) Bumps [mixin-deep](https://github.com/jonschlinkert/mixin-deep) from 1.3.1 to 1.3.2. - [Release notes](https://github.com/jonschlinkert/mixin-deep/releases) - [Commits](jonschlinkert/mixin-deep@1.3.1...1.3.2) Signed-off-by: dependabot[bot] <support@github.com> commit e81a77a Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri Aug 30 23:33:36 2019 +0200 chore: bump lodash.template from 4.4.0 to 4.5.0 (typeorm#4416) Bumps [lodash.template](https://github.com/lodash/lodash) from 4.4.0 to 4.5.0. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.4.0...4.5.0) Signed-off-by: dependabot[bot] <support@github.com> commit 00d46e1 Author: JB Reefer <james.reefer@gmail.com> Date: Sat Aug 24 05:08:26 2019 -0400 docs: Update using-cli.md (typeorm#4618) Fix typo commit d1594f5 Author: Abhijeet Chakraborty <16278759+abhijeet1403@users.noreply.github.com> Date: Wed Aug 14 14:01:48 2019 +0530 fix: resolve issue with conversion string to simple-json (typeorm#4476) Closes: typeorm#4440 commit c321562 Author: Arseny Yankovsky <ArsenyYankovsky@users.noreply.github.com> Date: Tue Aug 13 10:12:25 2019 +0200 feat: Aurora Data API (typeorm#4375) * Initial POC Implementation * Initial POC Implementation * Implemented an interface transformation between typeorm and data api so most of the queries should work, added some tests * Fixed lint errors * Fixed a regex and added some tests on query transformation * Move out to a separate repo * Bumped aurora driver version to latest * Bumped aurora driver version to latest * Delegate transactions to the driver * Delegate transactions to the driver * WIP * WIP * Bump the aurora driver version * Bump the aurora driver version * Fixed aurora driver version * removed unused entity commit b4e9cf0 Author: Marcos <holamarcosaurios@gmail.com> Date: Wed Aug 7 11:35:11 2019 +0200 docs: update typo in separating-entity-definition.md Typo in line 190: "Categeory" changed to "Category" commit 9930283 Author: Charlie Cruzan <35579283+cruzach@users.noreply.github.com> Date: Wed Aug 7 05:33:42 2019 -0400 docs: update outdated Expo SQLite link * update Expo SQLite link * docs: fix expo-example repo link commit 5e00e81 Author: Humberto Villalpando <43765965+betov18x@users.noreply.github.com> Date: Wed Jul 31 06:47:36 2019 -0500 chore: Update README.md Added some links for nativescript examples with vue and angular commit 10a5182 Author: markamPL <markam.tg@gmail.com> Date: Wed Jul 24 09:00:47 2019 +0200 docs: update typo in using-ormconfig.md Typo in .env configuration: .*js changed to *.js commit b6d6278 Author: Liau Jian Jie <liaujianjie@gmail.com> Date: Sun Jul 21 22:10:53 2019 +0800 feat: add multi-dimensional cube support for PostgreSQL (typeorm#4378) commit e12479e Author: Abdoulaye K. Traoré <abdoulayekt@hotmail.com> Date: Sat Jul 13 18:57:16 2019 +0200 feat: log files loaded from glob patterns (typeorm#4346) This new feature logs the files that are loaded using the glob patterns to aid in debugging. Closes: typeorm#4162 commit c8dbf09 Author: Mike Harris <mharris717@gmail.com> Date: Wed Jul 10 09:17:31 2019 -0400 docs: Fix Typo (typeorm#4412) commit a858de1 Author: Vlad Poluch <vlad.poluch@gmail.com> Date: Mon Jul 1 16:01:09 2019 +0200 Revert "docs: fix typographical error in faq (typeorm#4321)" (typeorm#4380) This reverts commit 117185b. commit 39a8e34 Author: Benjamin Dobell <benjamin.dobell+github@glassechidna.com.au> Date: Mon Jul 1 01:18:55 2019 +1000 feat: Added support for DISTINCT queries (typeorm#4109) commit c8a9ea0 Author: Max Sommer <kontakt@maxsommer.de> Date: Sun Jun 30 17:12:07 2019 +0200 docs: add explanation ManyToMany with custom properties (typeorm#4308) * Add explanation ManyToMany with custom properties Since I myself ran into the issue and had to rewrite my code based on this solution I hope it may be helpful for others as well 😊 * Update explanation with more relateable example Update explanation with abstract example to be more relateable Add cross link to faq * Update phrasing, unnecessary explanation and code Remove unnecessary explanation upfront mentioning FAQ Update phrasing of main explanatory paragraph to be more precise Update code example to work correctly commit 79bf9f7 Author: Kononnable <kononnable@gmail.com> Date: Sun Jun 30 17:11:06 2019 +0200 build: node version upgrade in travis (typeorm#4312) * upgrading node CI tested versions * sqlite3 version upgrade commit a6d7ba2 Author: Leonardo Falk <leonardofalk@users.noreply.github.com> Date: Sun Jun 30 12:07:19 2019 -0300 fix: add SaveOptions and RemoveOptions into ActiveRecord (typeorm#4318) * Adding SaveOptions to BaseEntity#save * Adding RemoveOptions to BaseEntity#remove commit 684ffd1 Merge: 6429ccd 6a1206e Author: Umed Khudoiberdiev <pleerock.me@gmail.com> Date: Sun Jun 30 14:39:21 2019 +0300 Merge pull request typeorm#4376 from typeorm/revert-4306-fix-4291 Revert "fix: improve sql.js v1.0 support in browser environment" commit 6a1206e Author: Umed Khudoiberdiev <pleerock.me@gmail.com> Date: Sun Jun 30 14:38:54 2019 +0300 Revert "fix: improve sql.js v1.0 support in browser environment" commit 6429ccd Merge: 117185b ed87e34 Author: Umed Khudoiberdiev <pleerock.me@gmail.com> Date: Sun Jun 30 14:00:46 2019 +0300 Merge pull request typeorm#4306 from michaelbromley/fix-4291 fix: improve sql.js v1.0 support in browser environment commit 117185b Author: SriNath <12288245+SrChip15@users.noreply.github.com> Date: Wed Jun 19 09:13:47 2019 -0400 docs: fix typographical error in faq (typeorm#4321) change "typedi" to "typed" in how to use TypeORM with a dependency injection tool question commit ed87e34 Author: Michael Bromley <michael@michaelbromley.co.uk> Date: Tue Jun 18 10:25:35 2019 +0200 Add more documentation on using sql.js in the browser commit f7bcd8f Author: Michael Bromley <michael@michaelbromley.co.uk> Date: Tue Jun 18 10:05:04 2019 +0200 Fix sql.js v1.0 support in browser
@ArsenyYankovsky did you manage to find out any solution for "Communications link failure" error? It seems other people are experiencing it as well -> www.jeremydaly.com/aurora-serverless-data-api-a-first-look/ and forums.aws.amazon.com/thread.jspa?messageID=915356 . |
@hakimio the communications failure is just the serverless database starting up. In production I always set the minimum ACU to 1 and instruct it to not pause the database. Then this problem doesn’t happen. In development environments users just need to be aware this is the case and wait about a minute for the database to start. I have found this acceptable thus far. Of course based on the environment you can tweak the idle timeout. |
@hakimio Exactly what @TreeMan360 said. If you're running a background task you might want to hit the db with a simple query just to check if it's up or down and then wait some time to allow the cluster to wake up. |
@ArsenyYankovsky I'm running into this error intermittently: |
@gitsome Not sure I understand you correctly, but yes, try to run the latest version of the driver available in npm. I have nightly tests running with the latest versions of TypeORM and the driver so a simple scenario like that is supposed to be working. |
Update, @ArsenyYankovsky. When I switch from using |
@ArsenyYankovsky can you update README for data api implementation ? |
@ArsenyYankovsky right now we have |
@pleerock I think it's a good idea to make the name more obvious. Please let me know when it's done so I can set up tests in the driver against the new branch. |
@ArsenyYankovsky okay, thanks for the response. This change will be done in 0.3.0 |
Motivation
Connection management is hard in the serverless environments. To address this issue AWS recently released Data API for their Aurora Serverless database.
Current State
Moved all Aurora Data API related code to a separate repo here. Most of the issues should be resolved there.
Things that work
TODO
Is going to be tracked in the drvier repo