Added chunksize argument to to_sql

[artemyk]

Large dataframes may fail to write to a database in one go due to packet size errors . This add a chunksize parameter which will write the dataframe to the database in batches. See #7347 .

Notice that there is quite a bit of code duplication between the SQLAlchemy and legacy APIs. This could be factored out if desired at the cost of a few private methods.

Closes #7347

[jorisvandenbossche]

Thanks for the PR! Looking good at first sight, will try to do a more detailed review tomorrow.

[artemyk]

Just something to consider --- it might be better to strip out the transaction handling code and just document the fact that batch writing is non-atomic. It seems like sqlite, at least, does not support transactions for DDL statements like CREATE TABLE ... , so the transaction code adds cruft but does not fulfill its intended purpose.

[jorisvandenbossche]

What is the advantage of working with a Session here (and do the commit and rollback manually, opposed to what it was before)?
I am not yet fully familiar with all those SQLAlchemy features, but I thought we did, untill now, just used the 'core' with engines and connections, and not the higher-level abstraction with ORM.

[artemyk]

@jorisvandenbossche thanks for the comments, I implemented the suggested fixes. The reason for using Session() was some weirdness with sqlalchemy 0.7 (specifically, Engine does not support contextmanager), but now I got around it using only low-level elements.

[jorisvandenbossche]

why is this line added? I don't see keys used further in the function (maybe copied by mistake from the sqlalchemy version?)

[jorisvandenbossche]

About the version, if you need a newer version for something, we always can consider setting a higher minimum required version. In any case, we will have to raise the bar to 0.7.4 to support schema's.

For the PR, added two comments. Looking good for the rest! And further:

• needs a whatsnew entry (in enhancements)
• can you add also a small section in the docs? (in the "Writing DataFrames" section (http://pandas.pydata.org/pandas-docs/stable/io.html#writing-dataframes)) For example add in short something like "If you the connection times out, you can solve this by using the chunksize .."
• squash the commits

Thanks a lot!

[artemyk]

Let me know regarding changing sqlalchemy required version vs. leaving transaction code as-is and I will make changes if needed and then squash the commits. Thanks greatly!

[jorisvandenbossche]

formatting note: better use double quotes like chunksize (then it renders like code)

+ can you add a link to the issue number (see how it is done for the others)

[jorisvandenbossche]

apart form the formatting comments, and the one comment above on keys (#8062 (comment)), I think this is ready to squash and merge

@danielballan @hayd @mangecoeur of someone else wants to give this a look, certainly welcome

[artemyk]

Removed keys (good catch), fixed doc changes, and squashed the commits.

[jorisvandenbossche]

Thanks a lot!