Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

skiprows in read_csv not working? #505

Closed
solomon-negusse opened this issue Dec 19, 2011 · 4 comments

Comments

@solomon-negusse
Copy link

commented Dec 19, 2011

Hi Wes,
I'm trying to read in a csv file with few header lines at the beginning of the file starting w/ a '#'. I tried to use skiprows specifying a list of the first n rows to skip as [0,1,2,3,..,n] but I keep getting Index has duplicates ['#'] error. Is this a bug or I'm doing something wrong? Also, will this be the best place to post such issues?
I'm using pandas v 0.6.0 with python 2.7.2
Thanks for the great work you are doing w/ pandas. I'm finding it very useful and saves me a lot of time analyzing time series data.

-Solomon

@wesm

This comment has been minimized.

Copy link
Member

commented Dec 19, 2011

Could you construct a self-contained example illustrating the problem?

@solomon-negusse

This comment has been minimized.

Copy link
Author

commented Dec 19, 2011

Not sure of the posting protocol to follow. Here is few lines from the file (not very clean) that I tried to read using:

flow_series = pandas.read_csv(pd_file, sep=',',index_col=0, skiprows=[0,1,2,3,4,5,6], parse_dates=True)

#,Dow River Water System Operations,,,,,,,,January,2008 ,,,,Date Time of Report,,,,15-Dec ,04:43 PM ,,,,,,,,Page 1 of 3,,Dow River Water System Operations,,,,,,,,January,2008 ,,,,Date Time of Report,,,,15-Dec ,04:43 PM ,,,,,,,Page 2 of 3,,Dow River Water System Operations,,,,,,,,January,2008 ,,,,Date Time of Report,,,,15-Dec ,04:43 PM ,,,,,,,Page 3 of 3,,,,,,,,,,,,,
#,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#,MO/YR,Harris Reservoir,,,,,,,Lake Jackson Pump Station,,,Brazoria Reservoir,,,,,,,Buffalo Camp Bayou,,,,BWA Usage,,Plant B Water Flow,,,,MO/YR,River Flow By USGS ,,Water Master,,River Chlorides,,,Harris Pump Station,,,,,Brazoria Pump Station,,,,,Lake Jackson Pump Station,,,,,,, NOTES,,Brazos River Basin Permit Data,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#,Jan-07, RIV SLT, SILT, TOT IN, LEVEL , DISCH, DISCH, DISCH, OC LVL, OC DAM, DISCH, RIV SLT , SILT, TOT IN, LEVEL , DISCH , DISCH , DISCH,Diversion Gates,,, SPLWAY, TOTAL, TOTAL, PLT RATE, TOTAL, RETURNS,,Jan-07,GAUGE, AMOUNT, REQUEST, GRANTED, WEDGE, @braz, @harris, P-1, P-1 , P-2, P-2 , TOTAL, P-1, P-1, P-2, P-2, TOTAL,P-2,P-3,P-4,P-5,P-6, TO PLT, TOTAL,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#, DATE, %, ac-ft, ac-ft, ft, gpm, hrs, ac-ft, ft, gpm, ac-ft, %,CY, ac-ft, ft , gpm , hrs , ac-ft, gpm, ac-ft,ft, ac-ft, Mgal/d, ac-ft, gpm, ac-ft, gpm,,DATE, LOC, ac-ft , ac-ft, ac-ft, Riv Mi, ppm, ppm, gpm, hrs, gpm, hrs, MMgal, gpm, hrs, gpm, hrs, MMgal, hrs, hrs, hrs, hrs, hrs, avg gpm, MMgal,,,,Permit Number,,Permit Date,,Authorized Diversion,,,LOCATION,,,,,COMMENTS,"- ALL PERMITS FOR INDUSTRIAL, MANUFACTURING, AND MUNICIPAL USE.",,,,,,,,,,,,,,,,,,,,,,,,
#,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
1/1/2008, 1,0.00 ,0.00 ,220.96 ,41.38 ,37205 ,24.00 ,164.42 ,4.60 ,4075 ,220.96 ,,0 ,0.00 ,30.50 ,45302 ,24.00 ,200.20 ,51970 ,229.67 ,12.64 ,0.00 ,,0.00 ,101970 ,450.62 ,,, 1,ROSH,4482.71 ,,,,,88 ,0 ,0.0 ,50000 ,24.0 ,72.000 ,0 ,0.0 ,0 ,0.0 ,0.000 ,0.0 ,24.0 ,24.0 ,0.0 ,0.0 ,50000 ,72.000 ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
1/2/2008, 2,0.00 ,0.00 ,220.96 ,41.40 ,37205 ,24.00 ,164.42 ,4.60 ,4075 ,220.96 ,,0 ,0.00 ,30.35 ,45302 ,24.00 ,200.20 ,51593 ,228.00 ,12.64 ,0.00 ,4894.00 ,15.02 ,98194 ,433.94 ,0 ,, 2,ROSH,4105.84 ,,,,,88 ,0 ,0.0 ,50000 ,24.0 ,72.000 ,0 ,0.0 ,0 ,0.0 ,0.000 ,0.0 ,24.0 ,24.0 ,0.0 ,0.0 ,50000 ,72.000 ,,,,1145-C,,6-6-33,," 20,000 a",c-ft/year,,Brazos River/Harris Reservoir,,,,,Purchased rights from south Texas Water company in 1942,,,,,,,,,,,,,,,,,,,,,,,,,
1/3/2008, 3,0.00 ,0.00 ,220.96 ,41.40 ,37205 ,24.00 ,164.42 ,4.60 ,4075 ,220.96 ,,0 ,0.00 ,30.20 ,45302 ,24.00 ,200.20 ,51274 ,226.59 ,12.58 ,0.00 ,6295.00 ,19.32 ,96902 ,428.23 ,0 ,, 3,ROSH,3808.32 ,,,19.0 ,,88 ,0 ,0.0 ,50000 ,24.0 ,72.000 ,0 ,0.0 ,0 ,0.0 ,0.000 ,0.0 ,24.0 ,24.0 ,0.0 ,0.0 ,50000 ,72.000 ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
1/4/2008, 4,0.00 ,0.00 ,220.96 ,41.45 ,37205 ,24.00 ,164.42 ,4.60 ,4075 ,220.96 ,,0 ,0.00 ,30.15 ,45302 ,24.00 ,200.20 ,50042 ,221.15 ,12.52 ,0.00 ,5973.00 ,18.33 ,95894 ,423.77 ,0 ,, 4,ROSH,3629.80 ,,,,,88 ,0 ,0.0 ,50000 ,24.0 ,72.000 ,0 ,0.0 ,0 ,0.0 ,0.000 ,0.0 ,24.0 ,24.0 ,0.0 ,0.0 ,50000 ,72.000 ,,,,1345,,2-19-43, *,"150,000 a",c-ft/year,,Brazos River/Harris Reservoir,,,,,Authourizes use of Oyster creek to convey water from Harris Reservoir to Lake Jackson Pump Station,,,,,,,,,,,,,,,,,,,,,,,,,
1/5/2008, 5,0.00 ,0.00 ,220.96 ,41.46 ,37205 ,24.00 ,164.42 ,4.60 ,4075 ,220.96 ,,0 ,0.00 ,30.03 ,39321 ,24.00 ,173.77 ,49192 ,217.39 ,12.55 ,0.00 ,,0.00 ,99192 ,438.35 ,,, 5,ROSH,3669.47 ,,,,,88 ,0 ,0.0 ,50000 ,24.0 ,72.000 ,0 ,0.0 ,0 ,0.0 ,0.000 ,0.0 ,24.0 ,24.0 ,0.0 ,0.0 ,50000 ,72.000 ,,,,,,,," 60,000 a",c-ft/year,,Oyster Creek/Lake Jackson Pump Station,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

Let me know if you'd rather have me send a copy of the file.

@wesm

This comment has been minimized.

Copy link
Member

commented Dec 21, 2011

Fixed in the above commit. Needed a bit of refactoring to do things properly. thanks for the report

@wesm wesm closed this Dec 21, 2011

@solomon-negusse

This comment has been minimized.

Copy link
Author

commented Dec 21, 2011

thanks for the great tool.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.