Allow non-string load/save parameters #276
Conversation
Additional parameters to DataFrameReader.load() and DataFrameWriter.save() are passed to the file-type specific reader or writer types. These parameters can be of any type.
|
First of all thanks for your work. Additionally to my comment to the issue (#275 (comment)) I am not convinced here. Some broad type ( This however is quite brittle as it depend on both details of implementation ( Additionally I try to avoid |
| @@ -54,7 +54,7 @@ class DataFrameWriter(OptionUtils): | |||
| def sortBy(self, col: TupleOrListOfString) -> DataFrameWriter: ... | |||
| def save(self, path: Optional[str] = ..., format: Optional[str] = ..., mode: Optional[str] = ..., partitionBy: Optional[List[str]] = ..., **options: Any) -> None: ... | |||
| def insertInto(self, tableName: str, overwrite: Optional[bool] = ...) -> None: ... | |||
| def saveAsTable(self, name: str, format: Optional[str] = ..., mode: Optional[str] = ..., partitionBy: Optional[List[str]] = ..., **options: str) -> None: ... | |||
| def saveAsTable(self, name: str, format: Optional[str] = ..., mode: Optional[str] = ..., partitionBy: Optional[List[str]] = ..., **options: Any) -> None: ... | |||
There was a problem hiding this comment.
In general I still haven't made my mind if I am fine with making this annotation less strict. But just for the sake of argument - Optional[Union[bool, int, float str]] (extended #275 (comment)) is something that is up for discussion as:
- There is reasonable justification for it - there are already
optionmethods withBool,Double,Long,String. - These are rather unlikely to be extended and / or modified by the final user, so string representation is predictable.
- Smaller set of types is usually preferred, as it helps catching some naive mistakes.
- In case of unlikely event that this project is merged with upstream and inlined, we really want to avoid
Any(stubs don't validate against the annotated source, so it doesn't matter much at the moment).
|
@zero323 I'll fix this up as suggested as soon as I get a chance. |
|
Do you still plan to work on this @mark-oppenheim. If not, I might take a look this weekend, if I find a spare moment. |
|
Superseded by #363 |
Resolves #273
Additional parameters to DataFrameReader.load() and DataFrameWriter.save()/.saveTable() are passed to the file-type specific reader or writer types. These parameters can be of any type.