Skip to content

Commit

Permalink
firebase2graphql: improve readme structure, update git and add more t…
Browse files Browse the repository at this point in the history
…ests (#666)
  • Loading branch information
wawhal authored and shahidhk committed Oct 8, 2018
1 parent 9a9a5b9 commit 4f15e98
Show file tree
Hide file tree
Showing 8 changed files with 259 additions and 31 deletions.
204 changes: 182 additions & 22 deletions README.md
Expand Up @@ -8,6 +8,25 @@ A CLI tool to help you try realtime GraphQL on your firebase data. It takes data

![GIF](https://graphql-engine-cdn.hasura.io/assets/firebase2graphql/demo.gif)

## Contents

- [Getting started](#quick-start)
- [Installation](#installation)
- [Usage](#usage)
- [Command](#command)
- [Usage comparison](#usage-comparison---firebase-sdk-vs-graphql)
- [Authentication](#authentication)
- [Next steps](#next-steps)
- [More information](#more-information)
- [Working](#working)
- [Normalization](#normalization)
- [Automatic](#automatic)
- [Manual](#manual)
- [Duplicates](#duplicates)
- [Overwrite](#overwrite)
- [Feedback](#feedback)


## Quick start

1. Quickly get the GraphQL Engine running by clicking this button:
Expand Down Expand Up @@ -177,8 +196,6 @@ Check out [next steps](#next-steps).

## Installation

### CLI

```bash
npm install -g firebase2graphql
```
Expand Down Expand Up @@ -215,22 +232,7 @@ firebase2graphql URL [flags]
- `-v --version`: show CLI version
- `-h, --help`: show CLI help

## Next steps

Once you have imported your data, it is recommended that you make it production ready.

1. Normalize the data by [removing duplicates](#duplicates).
2. Explore the GraphQL Engine Console to play with things such as

- [Relationships](https://docs.hasura.io/1.0/graphql/manual/schema/relationships/index.html)
- [Permissions](https://docs.hasura.io/1.0/graphql/manual/auth/index.html)
- Using SQL
- [Set up async business logic using event triggers](https://docs.hasura.io/1.0/graphql/manual/event-triggers/index.html)
- [Create new tables](https://docs.hasura.io/1.0/graphql/manual/schema/basics.html)

3. Set appropriate permissions. GraphQL Engine comes with [fine grained control layer](https://docs.hasura.io/1.0/graphql/manual/auth/index.html) that can be integrated with any standard Auth provider.

## Usage Comparison - Firebase SDK vs GraphQL
## Usage comparison - Firebase SDK vs GraphQL

A typical query to do a single read from the database using [Firebase SDK](https://firebase.google.com/docs/reference/), (javascript) would look something like:

Expand Down Expand Up @@ -275,7 +277,26 @@ mutation {
}
```

## Things to know about implementation
## Authentication

Hasura can be integrated with most standard Authentication mechanisms including Firebase and Auth0. [Check out the authentication docs here](https://docs.hasura.io/1.0/graphql/manual/auth/index.html).

## Next steps

Once you have imported your data, it is recommended that you make it production ready.

1. [Normalize the data](#normalization)
2. Explore the GraphQL Engine Console to play with things such as

- [Relationships](https://docs.hasura.io/1.0/graphql/manual/schema/relationships/index.html)
- [Permissions](https://docs.hasura.io/1.0/graphql/manual/auth/index.html)
- Using SQL
- [Set up async business logic using event triggers](https://docs.hasura.io/1.0/graphql/manual/event-triggers/index.html)
- [Create new tables](https://docs.hasura.io/1.0/graphql/manual/schema/basics.html)

3. Set appropriate permissions. GraphQL Engine comes with [fine grained control layer](https://docs.hasura.io/1.0/graphql/manual/auth/index.html) that can be integrated with any standard Auth provider.

## More information

### Working

Expand All @@ -287,6 +308,8 @@ If you use the flag `--normalize`, the CLI finds out if the children tables are

### Normalization

#### Automatic

The CLI provides a flag called `--normalize` if you want to normalize your denormalized database.

A lot of guess-work is done by the CLI while normalizing the database. Here are some thing you need to know:
Expand All @@ -295,6 +318,143 @@ A lot of guess-work is done by the CLI while normalizing the database. Here are
2. Children tables are deleted if they are detected to be duplicates of some other root or child table.
3. In case of some children tables, when the data lacks a unique identifier, an extra unique field is added. In most cases, this field gets deleted while mergine a duplicate table with the original table.

#### Manual

In some cases, due to inconsistent field names or due to insufficient data, the CLI might not be able to normalize your database; but it makes sure that you do not lose any data.

In such cases, you can normalize the data yourself. Lets look at an example.

Consider this firebase database. This is the database of the official Firebase example app:

```json
{
"posts" : {
"-LMbLFOAW2q6GO1bD-5g" : {
"author" : "Eena",
"authorPic" : "https://lh4.googleusercontent.com/-vPOIBOxCUpo/AAAAAAAAAAI/AAAAAAAAAFo/SKk9hpOB7v4/photo.jpg",
"body" : "My first post content\nAnd body\nANd structure",
"starCount" : 0,
"title" : "My first post",
"uid" : "4UPmbcaqZKT2NdAAqBahXj4tHYN2"
},
"-LMbLIv6VKHYul7p_PZ-" : {
"author" : "Eena",
"authorPic" : "https://lh4.googleusercontent.com/-vPOIBOxCUpo/AAAAAAAAAAI/AAAAAAAAAFo/SKk9hpOB7v4/photo.jpg",
"body" : "AKsdjak\naklsdjaskldjklas\nasdklfjaklsdfjklsda\nasdklfjasklf",
"starCount" : 0,
"title" : "Whatta proaaa",
"uid" : "4UPmbcaqZKT2NdAAqBahXj4tHYN2"
}
},
"user-posts" : {
"4UPmbcaqZKT2NdAAqBahXj4tHYN2" : {
"-LMbLFOAW2q6GO1bD-5g" : {
"author" : "Eena",
"authorPic" : "https://lh4.googleusercontent.com/-vPOIBOxCUpo/AAAAAAAAAAI/AAAAAAAAAFo/SKk9hpOB7v4/photo.jpg",
"body" : "My first post content\nAnd body\nANd structure",
"starCount" : 0,
"title" : "My first post",
"uid" : "4UPmbcaqZKT2NdAAqBahXj4tHYN2"
},
"-LMbLIv6VKHYul7p_PZ-" : {
"author" : "Eena",
"authorPic" : "https://lh4.googleusercontent.com/-vPOIBOxCUpo/AAAAAAAAAAI/AAAAAAAAAFo/SKk9hpOB7v4/photo.jpg",
"body" : "AKsdjak\naklsdjaskldjklas\nasdklfjaklsdfjklsda\nasdklfjasklf",
"starCount" : 0,
"title" : "Whatta proaaa",
"uid" : "4UPmbcaqZKT2NdAAqBahXj4tHYN2"
}
}
},
"users" : {
"4UPmbcaqZKT2NdAAqBahXj4tHYN2" : {
"email" : "rishichandrawawhal@gmail.com",
"profile_picture" : "https://lh4.googleusercontent.com/-vPOIBOxCUpo/AAAAAAAAAAI/AAAAAAAAAFo/SKk9hpOB7v4/photo.jpg",
"username" : "Eena"
}
}
}
```

In case of this JSON, the CLI will generate the following tables:

```sql
users (
_id text not null primary key,
email text,
profile_picture text,
username text
)

posts (
_id text not null primary key,
title text,
body text,
starCount int,
author text,
authorPic text,
uid text
)

user_posts (
_id text not null,
_id_2 text not null,
title text,
body text,
starCount int,
author text,
authorPic text,
uid text
)
```

As we can see, this is not the most efficient of schemas.

- `posts(uid)` can be a foreign key referencing `users(_id)`. `posts(author)` and `posts(authorPic)` can be deleted if `users` and `posts` are related via a foreign key.
- `user_posts` table is obsolete if `posts` and `users` tables are deleted.

To normalize it, here are the steps you must follow:

1. Create the following foreign key constraints:

```sql
ALTER TABLE "posts" ADD CONSTRAINT "posts_users__uid" FOREIGN KEY ("uid") REFERENCES "users"("_id");

ALTER TABLE "posts" DROP COLUMN "author", DROP COLUMN "authorPic";

DROP TABLE "user_posts";
```

To create them, go to the Hasura Console and run the SQL in the `Data > SQL` section.

2. Create the relationships. In the console, go the desired table and add the relationship suggested based on the Foreign key. For example for the `posts` table in the above schema, the suggested relationship would be suggested like:

![suggested](assets/suggested-rel.png)

Click on `Add`. You can name it whatever you like. Lets call it `author` in this case.

![added](assets/added-rel.png)


Similarly, add the suggested relationships for other tables as well.


3. Once you have created relationships, you can start making fancy GraphQL queries like:

```graphql
query {
users (order_by: username_asc){
username
profile_picture
email
posts (where: { starCount: { _gte: 100}}){
title
body
starCount
}
}
}
```

### Duplicates

Expand All @@ -304,12 +464,12 @@ In such cases, you have three choices:

1. Use the API as such if you prefer the exact API.
2. Go to the UI Console and delete the duplicates and normalize the database as you feel fit.
3. Use the `--normalize` flag and rerun the migration. In this case, the CLI will detect duplicates and make appropriate relationships between root nodes. (This feature is experimental and needs more test cases to attain stability. Contributions are welcome)
3. Use the `--normalize` flag and rerun the migration. In this case, the CLI will detect duplicates and make appropriate relationships between root nodes. (This feature is experimental and needs more test cases to attain stability. Contributions are welcome)


### Overwrite

If your database already contains tables with the same name as the root fields of your JSON database, the command will fail. If you want to overwrite the database anyway, you should provide an additional flag "--overwrite".
If your database already contains tables with the same name as the root fields of your JSON database, the command will fail. If you want to overwrite the database anyway, you should provide an additional flag `--overwrite`.

## Feedback

Expand Down
Binary file added assets/added-rel.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/suggested-rel.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 3 additions & 3 deletions test/data-sets/fbUsers.js
@@ -1,10 +1,10 @@
module.exports = {
scores: {
f2g_test_scores: {
Rishi: 24,
Rikin: 26,
Tanmai: 27,
},
author: {
f2g_test_author: {
someone: {
one: {
name: 'Rishi',
Expand Down Expand Up @@ -83,7 +83,7 @@ module.exports = {
},
},
},
articles: {
f2g_test_articles: {
first: {
title: 'Rishis article',
body: "Rishi's article's body",
Expand Down
6 changes: 3 additions & 3 deletions test/data-sets/readme-example-1.json
@@ -1,5 +1,5 @@
{
"Articles": {
"f2g_Articles": {
"A1": {
"Title": "Title1",
"Body": "Body1",
Expand Down Expand Up @@ -55,7 +55,7 @@
}
}
},
"Authors": {
"f2g_test_Authors": {
"AT1": {
"Name": "AName1",
"Age": 11,
Expand All @@ -77,7 +77,7 @@
}
}
},
"Comments": {
"f2g_test_Comments": {
"C1": {
"Body": "Comment1",
"Author": {
Expand Down
1 change: 1 addition & 0 deletions test/test.sh
Expand Up @@ -5,4 +5,5 @@ else
F2G_LOG=0 ../bin/run $TEST_HGE_URL --access-key=$TEST_X_HASURA_ACCESS_KEY --db=./data-sets/chinook.json --overwrite --normalize && node verifyChinook.js
F2G_LOG=0 ../bin/run $TEST_HGE_URL --access-key=$TEST_X_HASURA_ACCESS_KEY --db=./data-sets/blog.json --overwrite --normalize && node verifyBlog.js
F2G_LOG=0 ../bin/run $TEST_HGE_URL --access-key=$TEST_X_HASURA_ACCESS_KEY --db=./data-sets/chinook_nested.json --overwrite --normalize && node verifyChinookNested.js
F2G_LOG=0 ../bin/run $TEST_HGE_URL --access-key=$TEST_X_HASURA_ACCESS_KEY --db=./data-sets/readme-example-1.json --overwrite --normalize && node verifyRE1.js
fi
6 changes: 3 additions & 3 deletions test/verifyChinookNested.js
Expand Up @@ -56,17 +56,17 @@ const verifyDataImport = () => {
}),
}
).then(() => {
console.log(colors.green('✔︎ data-sets/chinook.json: Test passed'));
console.log(colors.green('✔︎ data-sets/chinook_nested.json: Test passed'));
process.exit();
}).catch(() => {
process.exit();
});
} else {
console.log(colors.red('✖ data-sets/chinook.json: Test failed. Unexpected response.'));
console.log(colors.red('✖ data-sets/chinook_nested.json: Test failed. Unexpected response.'));
process.exit();
}
}).catch(e => {
console.log(colors.red('✖ data-sets/chinook.json: Test failed. Unexpected response.'));
console.log(colors.red('✖ data-sets/chinook_nested.json: Test failed. Unexpected response.'));
console.log(JSON.stringify(e, null, 2));

process.exit();
Expand Down
67 changes: 67 additions & 0 deletions test/verifyRE1.js
@@ -0,0 +1,67 @@
const {query} = require('graphqurl');
const fetch = require('node-fetch');
const colors = require('colors/safe');

const complexQuery = `
query {
f2g_test_Authors (order_by: Name_asc) {
_id
Name
f2g_Articles (order_by: Title_asc, where: { IsUnpublished: { _eq: true}}) {
Title
f2g_test_Comments (order_by: Date_asc) {
Body
Date
}
}
}
}
`;

const verifyDataImport = () => {
query({
query: complexQuery,
endpoint: `${process.env.TEST_HGE_URL}/v1alpha1/graphql`,
headers: {'x-hasura-access-key': process.env.TEST_X_HASURA_ACCESS_KEY},
}).then(response => {
if (
response.data &&
response.data.f2g_test_Authors[0].f2g_Articles.length === 0 &&
response.data.f2g_test_Authors[1].f2g_Articles[0].f2g_test_Comments[0].Body === 'Comment1'
) {
let sqlString = '';
['Articles', 'Authors', 'Comments'].forEach(t => {
sqlString += `drop table public."f2g_test_${t}" cascade;`;
});
fetch(
`${process.env.TEST_HGE_URL}/v1/query`,
{
method: 'POST',
headers: {'x-hasura-access-key': process.env.TEST_X_HASURA_ACCESS_KEY},
body: JSON.stringify({
type: 'run_sql',
args: {
sql: sqlString,
cascade: true,
},
}),
}
).then(() => {
console.log(colors.green('✔︎ data-sets/readme-example-1.json: Test passed'));
process.exit();
}).catch(() => {
process.exit();
});
} else {
console.log(colors.red('✖ data-sets/readme-example-1.json: Test failed. Unexpected response.'));
process.exit();
}
}).catch(e => {
console.log(colors.red('✖ data-sets/readme-example-1.json: Test failed. Unexpected response.'));
console.log(JSON.stringify(e, null, 2));

process.exit();
});
};

verifyDataImport();

0 comments on commit 4f15e98

Please sign in to comment.