New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LOAD data INFILE导入大列时出错 #1073

Closed
wangjingfei opened this Issue Aug 24, 2016 · 4 comments

Comments

Projects
None yet
2 participants
@wangjingfei

wangjingfei commented Aug 24, 2016

当导入数据的某一列较大时,报以下错误,请教如何设置bufferSize:

08/24 06:09:32.012  ERROR [$_NIOREACTOR-1-RW] (FrontendConnection.java:270) -load data error
com.univocity.parsers.common.TextParsingException: Error processing input: Length of parsed input (4097) exceeds the maximum number of characters defined in your parser settings (4096).
Hint: Number of characters processed may have exceeded limit of 4096 characters per column. Use settings.setMaxCharsPerColumn(int) to define the maximum number of characters a column can ha
ve
Ensure your configuration is correct, with delimiters, quotes and escape sequences that match the input format you are trying to parse
Parser Configuration: CsvParserSettings:
        Column reordering enabled=true
        Empty value=null
        Header extraction enabled=false
        Headers=null
        Ignore leading whitespaces=true
        Ignore trailing whitespaces=true
        Input buffer size=1048576
        Input reading on separate thread=true
        Line separator detection enabled=false
        Maximum number of characters per column=4096
        Maximum number of columns=512
        Null value=null
        Number of records to read=all
        Parse unescaped quotes=true
        Row processor=none
        Selected fields=none
        Skip empty lines=trueFormat configuration:
        CsvFormat:
                Comment character=#
                Field delimiter=,
                Line separator (normalized)=\n
                Line separator sequence=\n
                Quote character="
                Quote escape character=quote escape
@magicdoom

This comment has been minimized.

Show comment
Hide comment
@magicdoom

magicdoom Aug 26, 2016

Contributor

1.6已经修改参数

Contributor

magicdoom commented Aug 26, 2016

1.6已经修改参数

@wangjingfei

This comment has been minimized.

Show comment
Hide comment
@wangjingfei

wangjingfei Aug 29, 2016

非常感谢!但是,我看到Commit中把setMaxCharsPerColumn数值写死了,不考虑使用配置文件加载这两个参数吗?

wangjingfei commented Aug 29, 2016

非常感谢!但是,我看到Commit中把setMaxCharsPerColumn数值写死了,不考虑使用配置文件加载这两个参数吗?

@magicdoom

This comment has been minimized.

Show comment
Hide comment
@magicdoom

magicdoom Aug 29, 2016

Contributor

已经改的比较大了 一般也够用了 以后或许考虑配置

Contributor

magicdoom commented Aug 29, 2016

已经改的比较大了 一般也够用了 以后或许考虑配置

@magicdoom magicdoom closed this Aug 29, 2016

@wangjingfei

This comment has been minimized.

Show comment
Hide comment
@wangjingfei

wangjingfei Aug 30, 2016

好的。正常情况下一行不超过64k也够了,出现long text这种变态的就不够了

wangjingfei commented Aug 30, 2016

好的。正常情况下一行不超过64k也够了,出现long text这种变态的就不够了

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment