Add antlr grammar and visitor for postgresql #416

NQPhuc · 2023-08-04T05:57:30Z

Summary

Add postgresql parser using antlr grammar (based on: https://github.com/antlr/grammars-v4/tree/master/sql/postgresql)
The old postgres parser can still be used through: sql2dbml --postgres-legacy
Add AST classes to unified parsers output. These class can be used to build the model_structure.
Implement PostgresASTGen visitor to generate the AST from the parse tree. Note that not all nodes will be visited in our visitor, only the nodes related to model_structure and thus DBML is.
Fix an error in dbml parser where index pk setting couldn't be used with other settings such as name
Update tests.
Note: Files in ANTLR/parsers/postgresql are auto-generated and can be safely ignored.

Issue

Lasting Changes (Technical)

Replace all jest.setTimeout() with test timeout 3rd argument, since the previous one has stop working in jest v29. Jest 27: jest.setTimeout has no effect when used in async test jestjs/jest#11543 (comment)
For column type define in schemaName.typeName format, if the corresponding enum can't be found, we now view it as a custom type of the column instead of throwing error.

Checklist

Please check directly on the box once each of these are done

Documentation (if necessary)
Tests (integration test/unit test)
Integration Tests Passed
Code Review

huyphung1602 · 2023-08-10T05:00:32Z

We should add some tests here. For example, put a PostgreSQL to the parser and the compare the output with the expected output.

huyphung1602 · 2023-08-10T05:03:12Z

packages/dbml-core/src/parse/ANTLR/ASTGeneration/PostgresASTGen.js

+    for (let i =  1; i < ctx.getChildCount(); i++) {
+      r += ` ${ctx.getChild(i).getText()}`;
+    };


Can we use lodash functions here?

Updated in 15d016f. Although I did not use lodash, I replaced the for loop and made it functional by generating an array range of indices and then reduce it into a string.

huyphung1602 · 2023-08-10T05:08:10Z

packages/dbml-core/src/parse/ANTLR/ASTGeneration/PostgresASTGen.js

@@ -0,0 +1,941 @@
+/* eslint-disable class-methods-use-this */
+import _ from 'lodash';


Please import only the lodash functions that you are using. For example:

import { map } from 'lodash';

Fixed in 15d016f.

huyphung1602 · 2023-08-10T05:12:37Z

Note: Except for the 2 based class, files in ANTLR/parsers/postgresql are auto-generated and can be safely ignored.

It is not useful for the reviewer em. You should list out the files that are auto-generated and can be skip:

ignore_path/*
file_name_1
file_name_2

TBH, I do not understand what is '2 based class'.

packages/dbml-core/src/parse/ANTLR/ASTGeneration/index.js

packages/dbml-core/__tests__/parser/postgres-parse/output/cmd_comment.out.json

packages/dbml-core/src/parse/ANTLR/parsers/postgresql/PostgreSQLLexerBase.js

packages/dbml-core/src/parse/ANTLR/ASTGeneration/index.js

packages/dbml-core/src/parse/ANTLR/parsers/postgresql/PostgreSQLParserBase.js

…issues tests into existing files.

…deprecated jest.setTimeout().

huyphung1602 · 2023-08-30T09:47:29Z

packages/dbml-core/__tests__/importer/postgres_importer/input/general_schema.in.sql

+-- issue 324
+CREATE TABLE "public"."accounts" (
+  "id" integer GENERATED ALWAYS AS IDENTITY NOT NULL,
+  "domain" character varying(100),
+  "name" character varying(100),
+  "slug" character varying(10),
+  CONSTRAINT "accounts_pkey" PRIMARY KEY ("id")
+) WITH (oids = false);
+
+-- issue 281
+CREATE TABLE public.tests ( id bigint NOT NULL, type character varying DEFAULT 'testing'::character varying );
+
+-- issue 427
+CREATE TABLE public.users2 (
+    username character varying NOT NULL,
+    hashed_password character varying NOT NULL,
+    full_name character varying NOT NULL,
+    email character varying NOT NULL,
+    password_changed_at timestamp with time zone DEFAULT '0001-01-01 00:00:00+00'::timestamp with time zone NOT NULL,
+    created_at timestamp with time zone DEFAULT now() NOT NULL
+);
+
+-- issue 248
+create table table1 (
+    field text not null,
+    field2 text not null,
+    primary key(field, field2)
+);
+
+create table if not exists table2 (
+    id int generated always as identity primary key,
+    field text,
+    field2 text,
+    foreign key (field, field2) references table1(field, field2)
+);
+
+-- issue 222
+CREATE TABLE test_table (
+	"name" varchar NOT NULL,
+	id varchar NULL,
+	linked_name varchar NOT NULL,
+	CONSTRAINT test_table_pk PRIMARY KEY ("name", id),
+	CONSTRAINT test_table_fk FOREIGN KEY (linked_name,id) REFERENCES test_table("name",id)
+);


Please add the issue links inside the code nha em

Added in 5acb21a

huyphung1602 · 2023-08-30T09:50:00Z

packages/dbml-core/src/parse/ANTLR/ASTGeneration/postgres/PostgreSQLLexerBase.js

+}
+
+// helper functions required by the generated grammar
+// To avoid error, we declare their signature and import them in the generated PostgreSQLLexer.js


You should mention that these methods were defaulted in C# but they were not defaulted in Javascript. Do these methods affect the result of the parser?

Updated the comment in 5acb21a.

Do these methods affect the result of the parser?

According to my testing, no.

huyphung1602

LGTM. However, because it supports more PostgreSQL syntaxes, performance may suffer. Please assist in benchmarking the parser.

nguyenalter · 2023-08-31T03:53:50Z

Please update the documentation page to include your changes related to the CLI and JS Module

NQPhuc added 6 commits August 4, 2023 12:57

Add antlr grammar and visitor for postgresql

e55d32b

remove yarn.lock

958dbbf

fix test

a76a0f6

Remove un-needed files

0e37031

Fix tests

c66168b

Clean up code

11365fe

NQPhuc marked this pull request as ready for review August 7, 2023 11:08

NQPhuc requested review from huyphung1602, nguyenalter and phuongduyphan August 7, 2023 11:08

chore: update readme and fix typos

732ce20

huyphung1602 reviewed Aug 10, 2023

View reviewed changes

packages/dbml-core/src/parse/ANTLR/ASTGeneration/index.js Outdated Show resolved Hide resolved

nguyenalter reviewed Aug 10, 2023

View reviewed changes

TeaNguyen mentioned this pull request Aug 16, 2023

mssql ALTER TABLE #418

Open

NQPhuc added 12 commits August 21, 2023 11:26

Fix eslint

5aad86b

Throw error for un-supported format

c2e5fa1

Remove useless constructor

5f5183b

Update tests

30ade72

Update postgres issue tests. Fix identifier visit

e45f992

Add db_dump test. Fix importing error regard non-enum type. Move old …

78dc0e8

…issues tests into existing files.

Update test

7a1d156

Add test for issue 248

2825406

Fix pk index setting bug in dbml parser. Add issue 222 test. Replace …

a49bbd7

…deprecated jest.setTimeout().

Add custom error listener. Fix ignored test.

1e576b2

CLI: Add --postgres-legacy to import postgres with old parser.

c6667b0

Only import used lodash functions. replace for loop with reduce

15d016f

NQPhuc added 2 commits August 28, 2023 15:56

Move base class to different folders from auto-generated files

be253ef

Add missing config option for postgres legacy

a20267a

huyphung1602 reviewed Aug 30, 2023

View reviewed changes

huyphung1602 approved these changes Aug 30, 2023

View reviewed changes

NQPhuc added 5 commits September 5, 2023 17:22

Add benchmark

4be2549

Update comment

5acb21a

Update readme

778ed45

Fix typo in comment

59bdf53

Fix typo in docs

7f5e314

NQPhuc merged commit 16edf62 into master Sep 6, 2023
4 checks passed

This was referenced Sep 6, 2023

sql2dbml postgres is not able to able to understand casting #281

Closed

Postgres sql2dbml errors #277

Closed

Error with composite foreign keys when importing from PostgreSQL #248

Closed

[Postgres] Import fails on self referencing composite key #222

Closed

NQPhuc added PR: Bug Fix 🐛 A type of pull request used for changelog categories PR: Breaking Change 💥 A type of pull request used for changelog categories PR: New Feature 🚀 A type of pull request used for changelog categories labels Sep 6, 2023

NQPhuc mentioned this pull request Sep 7, 2023

support of jsonb in postgresql #340

Closed

mermetbt mentioned this pull request Sep 7, 2023

Postgresql parsing fail on function body: no viable alternative at input #436

Open

NQPhuc mentioned this pull request Sep 7, 2023

Composite PK from SQL loses order when translated to dbml #299

Closed

coderabbitai bot mentioned this pull request Apr 25, 2024

Release v3.4.2 #562

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add antlr grammar and visitor for postgresql #416

Add antlr grammar and visitor for postgresql #416

NQPhuc commented Aug 4, 2023 •

edited by TeaNguyen

Loading

huyphung1602 commented Aug 10, 2023

huyphung1602 Aug 10, 2023 •

edited

Loading

NQPhuc Aug 24, 2023 •

edited

Loading

huyphung1602 Aug 10, 2023

NQPhuc Aug 24, 2023

huyphung1602 commented Aug 10, 2023

huyphung1602 Aug 30, 2023

NQPhuc Sep 6, 2023

huyphung1602 Aug 30, 2023

NQPhuc Sep 6, 2023

huyphung1602 left a comment •

edited

Loading

nguyenalter commented Aug 31, 2023

		@@ -0,0 +1,941 @@
		/* eslint-disable class-methods-use-this */
		import _ from 'lodash';

Add antlr grammar and visitor for postgresql #416

Add antlr grammar and visitor for postgresql #416

Conversation

NQPhuc commented Aug 4, 2023 • edited by TeaNguyen Loading

Summary

Issue

Lasting Changes (Technical)

Checklist

huyphung1602 commented Aug 10, 2023

huyphung1602 Aug 10, 2023 • edited Loading

Choose a reason for hiding this comment

NQPhuc Aug 24, 2023 • edited Loading

Choose a reason for hiding this comment

huyphung1602 Aug 10, 2023

Choose a reason for hiding this comment

NQPhuc Aug 24, 2023

Choose a reason for hiding this comment

huyphung1602 commented Aug 10, 2023

huyphung1602 Aug 30, 2023

Choose a reason for hiding this comment

NQPhuc Sep 6, 2023

Choose a reason for hiding this comment

huyphung1602 Aug 30, 2023

Choose a reason for hiding this comment

NQPhuc Sep 6, 2023

Choose a reason for hiding this comment

huyphung1602 left a comment • edited Loading

Choose a reason for hiding this comment

nguyenalter commented Aug 31, 2023

NQPhuc commented Aug 4, 2023 •

edited by TeaNguyen

Loading

huyphung1602 Aug 10, 2023 •

edited

Loading

NQPhuc Aug 24, 2023 •

edited

Loading

huyphung1602 left a comment •

edited

Loading