Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support UTF-16 encodings #138

Closed
jimhester opened this issue Jun 20, 2019 · 0 comments
Closed

Support UTF-16 encodings #138

jimhester opened this issue Jun 20, 2019 · 0 comments
Labels
feature a feature request or enhancement readr 📖 Issues related to readr compatibility

Comments

@jimhester
Copy link
Collaborator

# This is the Byte Order Mark:
bom <- as.raw(c(255, 254))
# This is the text.
text <- "1\t2\n"

# Converted to UTF-16LE
text_utf16 <- iconv(text,from="UTF-8",to="UTF-16LE", toRaw = TRUE)[[1]]

# Write the BOM and the text to a file
tmp_file_name <- tempfile()
fd <- file(tmp_file_name, "wb")
writeBin(bom, fd)
writeBin(text_utf16, fd)
close(fd)

# read the file and use iconv to convert it to UTF-8
con <- file(tmp_file_name, "rb")
x <- readBin(con, "raw", n = 100)
vroom::vroom(iconv(list(x), from = "UTF-16LE", to = "UTF-8", toRaw = TRUE), col_names = FALSE)
#> Observations: 1
#> Variables: 2
#> dbl [2]: X1, X2
#> 
#> Call `spec()` for a copy-pastable column specification
#> Specify the column types with `col_types` to quiet this message
#> # A tibble: 1 x 2
#>      X1    X2
#>   <dbl> <dbl>
#> 1     1     2

Created on 2019-06-20 by the reprex package (v0.3.0)

So we should basically be able to do this iconv conversion on the fly if given a multi-byte encoding.

@jimhester jimhester added the feature a feature request or enhancement label Jun 25, 2019
@jimhester jimhester added the readr 📖 Issues related to readr compatibility label May 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature a feature request or enhancement readr 📖 Issues related to readr compatibility
Projects
None yet
Development

No branches or pull requests

1 participant