Optimize way to convert natural language to text

[suyashgautam]

I am going through 2 research papers.

1. https://arxiv.org/pdf/2208.10099.pdf
2. https://arxiv.org/pdf/2208.04415.pdf
   Will create a summary of the two to come up with an optimized approach.

[goyalpramod]

Heya there @suyashgautam no information has been provided regarding this issue since your last comment, is the issue still open? Is the project open to new ideas and methodologies? If so I would love to contribute in whatever manner I can. I can begin with making a summary of the mentioned papers.

[suyashgautam]

Hi @goyalpramod , I got busy with other dev issues and didn't get time to come back to this. I would suggest do not limit yourself to just these research papers try to read more papers to find out which model will be best for our use case.
Looking forward to you contribution

[goyalpramod]

@suyashgautam Thank you for the clarification, I will try finding some recent developments and literature work on the said subject. Should I give you my findings over Discord? or should I send it over here. I am also interested in the particular project, So will discord be the best place to discuss about it?

[suyashgautam]

@goyalpramod Yeah you can ping me on discord.

[goyalpramod]

This is the summarization for the paper linked below.
https://link.springer.com/article/10.1007/s00778-022-00776-8
Talks about different datasets, how WikiSQL and spider are the most popular one. WikiSQL is simple and does not contain most of the complex SQL queries. Spider is a complex cross domain dataset. There have been additional datasets developed taking inspiration from spider. But Spider still remains the most popular benchmark
Different comparison metrics, and how each has its own pros and cons. And which one is being used most popularly right now
Talks about the methodology of Text 2 SQL
Section 4 talks about how to find relation between different words and sql queries, But each talks about different model and how it achieves a specified task. I will skip this part and talk about the next part. Which talks about using pretrained Language models.
rest of section 4 talks about how we can fine tune transformer models.
section 5 summarises all the different models discussed so far
https://link.springer.com/article/10.1007/s00778-022-00776-8/figures/7 This is a good summary of all the developments in the past 5 years
Section 6 talks about complete developed systems and how each is different from each other
https://link.springer.com/article/10.1007/s00778-022-00776-8/tables/5 This can help have a highlevel look into the working methodology of different models
Section 7 and 8 are a high level explanation of everything that has been happening in the recent pass and a conclusion

The following is a summarization of some models that were went through
https://huggingface.co/juierror/text-to-sql-with-table-schema -> this open source model has been trained on the wikiSQL dataset
https://huggingface.co/juierror/flan-t5-text2sql-with-schema -> this is an improvement on that, it uses a denser network.
The person has fine tuned these transformers on the wikiSQL dataset. Now the wikiSQL dataset has a big downfall (even though it is one of the two most popular Text 2 SQL datasets available, the second one being Spider) That it has only simple queries, I.e no complex join, merge, nested SQL etc present in it. So if we use this dataset. We are bound to hit an upper limit on the performance of the model itself.
https://huggingface.co/LarkAI/bart_large_nl2sql -> this is a bart fine tuned model which improves upon the shared models, the previous models could not rank. i.e when given NL like input: question: get people name with age less 25 table: id, name, age
output: SELECT name FROM table WHERE age < 25
This is also trained on wikiSQL, so it lacks a scope for improvement
Spider has a lot more complex SQL queries and provides much more potential for modelling.

[ChakshuGautam]

Let's create a place where we can keep articulating and measuring these things. @goyalpramod can you come up with a process of saving all of these literature review outputs somewhere in the repo?

[goyalpramod]

Let's create a directory and we can store the different models we tried and the results in different .md files. I can create an initial readme.md describing how the evaluation of each model should be noted, so it becomes a standardized process that everyone can follow upon.

[ChakshuGautam]

Aligned on this. Please go ahead and raise a PR.

[goyalpramod]

@ChakshuGautam I have created the PR #34 kindly go through it. After you approve of it, I will add the information from my review into it.