Skip to content
This repository has been archived by the owner on Sep 1, 2022. It is now read-only.

Fix data conversion error on CourseWork #40

Closed
dchess opened this issue Apr 17, 2020 · 6 comments · Fixed by #44
Closed

Fix data conversion error on CourseWork #40

dchess opened this issue Apr 17, 2020 · 6 comments · Fixed by #44
Labels
bug Something isn't working

Comments

@dchess
Copy link
Collaborator

dchess commented Apr 17, 2020

When importing the CourseWork endpoint, the following error is occurring:
sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('42000', '[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Error converting data type nvarchar to float. (8114) (SQLExecDirectW)')

This is either coming from the topicId or one of the dueDate/dueTime fields.

@dchess dchess added the bug Something isn't working label Apr 17, 2020
@dchess
Copy link
Collaborator Author

dchess commented Apr 21, 2020

@zkagin The description field is being interpreted as a float by pandas initially and then string are failing to import

@dchess
Copy link
Collaborator Author

dchess commented Apr 21, 2020

@zkagin a hacky fix that is working is to add sorted on the course_ids

@zkagin
Copy link
Collaborator

zkagin commented Apr 23, 2020

@dchess I am unable to reproduce this bug running Courses + CourseWork on an MSSQL Docker container (and removing the sorted() fix). Can you provide more info on which column / data seems to be causing the error?

I'm going to do the fix for #39 and may defer to you on whether that fixes this bug as well, if you can consistently repro.

@dchess
Copy link
Collaborator Author

dchess commented Apr 23, 2020

@zkagin make sure you set your school year start date to 2019-07-01. Its coming from the description field of about the 3rd record in.

@zkagin
Copy link
Collaborator

zkagin commented Apr 23, 2020

@dchess I have that set up, but on both MSSQL and Postgres I'm not running into the error. Perhaps the order of courses is non-deterministic or dependent on which user is requesting the data.

Can you confirm that the submitted PR #44 solves this issue? It removes the sorted so it should otherwise break if it doesn't correctly fix it.

@dchess dchess linked a pull request Apr 23, 2020 that will close this issue
dchess pushed a commit that referenced this issue Apr 24, 2020
Fixes #39, #40

Parses the separate coursework dueDate and dueTime columns into a single date column. This should fix a bug where having no dueDate in the first batch of coursework led to a column with an incorrect data type.
dchess pushed a commit that referenced this issue Apr 24, 2020
Fixes #39, #40

Parses the separate coursework dueDate and dueTime columns into a single date column. This should fix a bug where having no dueDate in the first batch of coursework led to a column with an incorrect data type.
@dchess dchess closed this as completed Apr 24, 2020
@dchess
Copy link
Collaborator Author

dchess commented Apr 25, 2020

@zkagin This is still coming through on the latest branch. It's related to the description field in CourseWork being set as float rather than string by pandas from the first few records. We may need to explicitly set dtypes similar to how we are doing it for dates.

@dchess dchess reopened this Apr 25, 2020
@dchess dchess closed this as completed in f5bb35e Apr 27, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants