dumping large pandas dataframe (~ >10k rows) times out

[notacoat]

a check to see how large the array is and then some code that chunks it into <=10k sections would work. I wrote a function that does that and it is working fine.

[fzumstein]

Great - can you open a pull request or do want to paste the function here? although the issue will not be dataframe specific but also apply to lists and numpy arrrays (they are all being transformed into lists before transferred over).

[notacoat]

here's the function I wrote.

def dump_largeDF(df, startcell='A1', chunk_size=10000):
    if len(df) <= (chunk_size +1): Range(startcell,index=False).value = df
    else: # chunk df and and dump each (default is 10k)\n",
        c = re.match(r"([a-z]+)([0-9]+)", startcell, re.I); cL = c.group(1); cN = int(c.group(2));n=0
        for i in (df[rw:rw + chunk_size] for rw in xrange(0, len(df), chunk_size)):
            if n==0: Range(cL + str(cN+n),index=False).value = i; cN += chunk_size
            else: Range(cL + str(cN+n)).value = i.values; cN += chunk_size
            n+=1

[fzumstein]

See also: http://stackoverflow.com/questions/27212643/how-can-i-insert-a-dataframe-in-excel-using-xlwings-without-pywintypes-com-error?noredirect=1#comment42912461_27212643

[Zenadix]

The function posted by EmilyBarbour is bugged and causes an empty row to appear between chunks.

I modified the function and now it works fine. I posted it on StackOverflow, but I'm posting it here too for easy access.

# Dumps a large DataFrame in Excel via xlwings
# Does not include headers.
def dump_largeDF(df, startcell='A1', chunk_size=100000):
    if len(df) <= (chunk_size + 1):
        Range(startcell, index=False, header=False).value = df
    else: # Chunk df and and dump each
        c = re.match(r"([a-z]+)([0-9]+)", startcell, re.I) # A1
        row = c.group(1) # A
        col = int(c.group(2)) # 1
        for chunk in (df[rw:rw + chunk_size] for rw in
                      range(0, len(df), chunk_size)):
            print("Dumping chunk in %s%s" %(row, col))
            Range(row + str(col), index=False, header=False).value = chunk
            col += chunk_size

For me, a chunk size of 100k is fine, however you may change it depending on your needs.