New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Invalide byte sequence in UTF-8 #1

Closed
borisschapira opened this Issue Aug 4, 2016 · 30 comments

Comments

5 participants
@borisschapira
Copy link

borisschapira commented Aug 4, 2016

Hi, I have an error even before building, just executing jekyll doctor.

Error: could not read file /Users/boris/Projects/perso/blog/jekyll-private/_posts/fr/papa/2016/2016-07-09-et-le-feu-ca-brule/1.jpg: invalid byte sequence in UTF-8

The same error is produced when I try to serve in verbose mode, I got this:

       ...
       EntryFilter: excluded /scripts
       EntryFilter: excluded /tasks
       EntryFilter: excluded /vendor
           Reading: _posts/en/default/2005-01-01-typo-test/2005-01-01-typo-test.md
           Reading: _posts/en/default/2014/2014-03-13-conquering-ab-testing/2014-03-13-conquering-ab-testing.md
           ... 
           ...
           Reading: _posts/fr/papa/2016/2016-07-01-esperanto/2016-07-01-esperanto.md
           Reading: _posts/fr/papa/2016/2016-07-09-et-le-feu-ca-brule/1.jpg
             Error: could not read file /Users/boris/Projects/perso/blog/jekyll-private/_posts/fr/papa/2016/2016-07-09-et-le-feu-ca-brule/1.jpg: invalid byte sequence in UTF-8
           Reading: _posts/fr/papa/2016/2016-07-09-et-le-feu-ca-brule/2016-07-09-et-le-feu-ca-brule.md
           Reading: _posts/fr/papa/2016/2016-07-10-tonton-et-tatie/2016-07-10-tonton-et-tatie.md
           ...
           ...
          Reading: _confs/sensibilisation-web.md
           Reading: _confs/web-disruption-continue.md
         Rendering: 2016/index.html
  Pre-Render Hooks: 2016/index.html
  Rendering Liquid: 2016/index.html
  Rendering Markup: 2016/index.html
  Rendering Layout: 2016/index.html
         Rendering: _posts/fr/default/2016/2016-06-15-conseils-pour-les-etudiants/2016-06-15-conseils-pour-les-etudiants.md
  Pre-Render Hooks: _posts/fr/default/2016/2016-06-15-conseils-pour-les-etudiants/2016-06-15-conseils-pour-les-etudiants.md
  Rendering Liquid: _posts/fr/default/2016/2016-06-15-conseils-pour-les-etudiants/2016-06-15-conseils-pour-les-etudiants.md
  Rendering Markup: _posts/fr/default/2016/2016-06-15-conseils-pour-les-etudiants/2016-06-15-conseils-pour-les-etudiants.md
         Requiring: kramdown
         Rendering: _posts/fr/papa/2016/2016-06-26-la-blockchain-expliquee-a-un-enfant-de-cinq-ans/2016-06-26-la-blockchain-expliquee-a-un-enfant-de-cinq-ans.md
  Pre-Render Hooks: _posts/fr/papa/2016/2016-06-26-la-blockchain-expliquee-a-un-enfant-de-cinq-ans/2016-06-26-la-blockchain-expliquee-a-un-enfant-de-cinq-ans.md
  Rendering Liquid: _posts/fr/papa/2016/2016-06-26-la-blockchain-expliquee-a-un-enfant-de-cinq-ans/2016-06-26-la-blockchain-expliquee-a-un-enfant-de-cinq-ans.md
  Rendering Markup: _posts/fr/papa/2016/2016-06-26-la-blockchain-expliquee-a-un-enfant-de-cinq-ans/2016-06-26-la-blockchain-expliquee-a-un-enfant-de-cinq-ans.md
  Liquid Exception: undefined method `gsub' for nil:NilClass in _layouts/archives-year.html
jekyll 3.2.0 | Error:  undefined method `gsub' for nil:NilClass
@borisschapira

This comment has been minimized.

Copy link

borisschapira commented Aug 4, 2016

I don't know why Jekyll tries to parse the images. Maybe something's missing in my conf or maybe it's because of my scoping (https://github.com/borisschapira/jekyll/blob/master/_config.yml#L13).

@nhoizey

This comment has been minimized.

Copy link
Owner

nhoizey commented Aug 4, 2016

It looks like it's the same issue as here: jekyll/jekyll#5181 (comment)

I don't understand, I don't have this issue on my side.

@nhoizey nhoizey added the bug label Aug 4, 2016

@nhoizey

This comment has been minimized.

Copy link
Owner

nhoizey commented Aug 4, 2016

I have this:

defaults:
  -
    scope:
      path: ""
      type: posts
    values:
      layout: post
@nhoizey

This comment has been minimized.

Copy link
Owner

nhoizey commented Aug 4, 2016

I'm still using Jekyll 3.1.6, can you try downgrading to test?

@nhoizey

This comment has been minimized.

Copy link
Owner

nhoizey commented Aug 4, 2016

@borisschapira

This comment has been minimized.

Copy link

borisschapira commented Aug 4, 2016

Tested and reported on jekyll/jekyll#5181 (comment).
What's your Ruby version ?

@nhoizey

This comment has been minimized.

Copy link
Owner

nhoizey commented Aug 4, 2016

ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin15]

@borisschapira

This comment has been minimized.

Copy link

borisschapira commented Aug 5, 2016

Same version. OS X El Capitan (I'm not uselessly curious, I'm wondering why the behaviour is different on our two computers)?

@nhoizey

This comment has been minimized.

Copy link
Owner

nhoizey commented Aug 5, 2016

That's really strange indeed, and you're not the only one with this issue…

@borisschapira

This comment has been minimized.

Copy link

borisschapira commented Aug 5, 2016

Ok so I just sent you my folder and you encounter the issue too. So in fact, you're the only one not having it 😄

@nhoizey

This comment has been minimized.

Copy link
Owner

nhoizey commented Aug 5, 2016

Trying with a fresh install on my Mac, and no issue:

$ jekyll new utf8
New jekyll site installed in /Users/nhoizey/Dropbox/Personnel/Devs/_bac_a_sable/utf8.

$ cd utf8

$ echo "gem 'jekyll-postfiles'" >> Gemfile

$ bundle install
Fetching gem metadata from https://rubygems.org/
Fetching version metadata from https://rubygems.org/
Fetching dependency metadata from https://rubygems.org/
Resolving dependencies...
Using colorator 1.1.0
Using ffi 1.9.14
Using forwardable-extended 2.6.0
Using sass 3.4.22
Using rb-fsevent 0.9.7
Using kramdown 1.11.1
Using liquid 3.0.6
Using mercenary 0.3.6
Using rouge 1.11.1
Using safe_yaml 1.0.4
Using jekyll-postfiles 2.0.0
Using minima 1.0.1
Using bundler 1.12.5
Using rb-inotify 0.9.7
Using pathutil 0.14.0
Using jekyll-sass-converter 1.4.0
Using listen 3.0.8
Using jekyll-watch 1.5.0
Using jekyll 3.2.1
Bundle complete! 3 Gemfile dependencies, 19 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.

$ echo "gems:\n  - jekyll-postfiles" >> _config.yml

$ cp ../duty_calls.png ./_posts

$ bundle exec jekyll build
Configuration file: /Users/nhoizey/Dropbox/Personnel/Devs/_bac_a_sable/utf8/_config.yml
            Source: /Users/nhoizey/Dropbox/Personnel/Devs/_bac_a_sable/utf8
       Destination: /Users/nhoizey/Dropbox/Personnel/Devs/_bac_a_sable/utf8/_site
 Incremental build: disabled. Enable with --incremental
      Generating...
                    done in 0.413 seconds.
 Auto-regeneration: disabled. Use --watch to enable.

$ ls _site/jekyll/update/2016/08/05
duty_calls.png         welcome-to-jekyll.html

duty_calls.png has been copied in _site without any issue…

@borisschapira

This comment has been minimized.

Copy link

borisschapira commented Aug 5, 2016

I've created a test case: test_ut8_issue.tar.gz

@borisschapira

This comment has been minimized.

Copy link

borisschapira commented Aug 5, 2016

In my test case, I use a local installation with bundle, not using any global lib.
Have you tried?

@nhoizey

This comment has been minimized.

Copy link
Owner

nhoizey commented Aug 5, 2016

As you can see in the commands sequence, I initiated the project with my global Jekyll (3.2.1), but then used bundle to install and run commands.

@borisschapira

This comment has been minimized.

Copy link

borisschapira commented Aug 5, 2016

@RRMoelker

This comment has been minimized.

Copy link

RRMoelker commented Sep 2, 2016

I'm having the same issue, excluding the images in the _config works:

exclude:
  - _posts/**/*.jpg
  - _posts/**/*.png
@borisschapira

This comment has been minimized.

Copy link

borisschapira commented Sep 4, 2016

@RRMoelker

  • Issue is solved if you are consulting the post page
  • Issue is still there if you are trying to display the content of the post elsewhere, as the URL to the ressource is relative, not absolute.

capture d ecran 2016-09-04 a 21 21 14

@RRMoelker

This comment has been minimized.

Copy link

RRMoelker commented Sep 5, 2016

I was still getting the invalid byte sequence during Jekyll build. I'm using:

Ruby 2.3.0
jekyll-postfiles (2.0.0)
jekyll (3.2.0)

The problematic image was a .png in a sub folder _post/<some-post>/thisOne.png. Excluding the .png files worked for me. So as far I'm not experience any issues at the moment.

@nhoizey

This comment has been minimized.

Copy link
Owner

nhoizey commented Sep 5, 2016

Well, I didn't have the issue until today, with already 750 images…

One of these images, for example:
dareboost-de-98-a-94

And today, I added this image:
2004-09-26-catching-the-sun

And I get this when I try to build the site:

Configuration file: _config.yml
Configuration file: _config_credentials.yml
            Source: /Users/nhoizey/Dropbox/Personnel/Devs/nicolas-hoizey.com
       Destination: /Users/nhoizey/Dropbox/Personnel/Devs/nicolas-hoizey.com/_site
 Incremental build: disabled. Enable with --incremental
      Generating...
             Error: could not read file /Users/nhoizey/Dropbox/Personnel/Devs/nicolas-hoizey.com/_posts/2008/07/25-catching-the-sun/2004-09-26-Catching-the-sun.jpg: invalid byte sequence in UTF-8

I added other images too, today, without issue.

So , IMHO, there must be something in the image itself that causes the issue.

@DirtyF

This comment has been minimized.

Copy link
Contributor

DirtyF commented Jan 16, 2017

same thing with jekyll 3.3.1

@nhoizey

This comment has been minimized.

Copy link
Owner

nhoizey commented May 4, 2017

Just understood why I had this error with these two files: it's because the file names were starting with dates. 2004-09-26-Catching-the-sun.jpg for example.

I suppose Jekyll was messing with these files, thinking they were posts.

So… I don't have the issue anymore after renaming the image files.

@borisschapira @RRMoelker @DirtyF do you still have the issue?

@RRMoelker

This comment has been minimized.

Copy link

RRMoelker commented May 4, 2017

Good to hear that solved it for you @nhoizey. For me the exclude option works for now.

But these kinds of bugs are way to prevalent with Jekyll. Considering the time I have for the project, I'm quite committed to the tech. For others, maybe consider a more customizable library if you want to do anything custom (as in not jekyll, I do not have any issue with the jekyll-postfiles library).

@nhoizey

This comment has been minimized.

Copy link
Owner

nhoizey commented May 4, 2017

@RRMoelker I don't really understand what you mean with "a more customizable library", sorry.

@RRMoelker

This comment has been minimized.

Copy link

RRMoelker commented May 4, 2017

Got into a bit of a rant. What I mean is that for anything more than the Jekyll standard blog, I don't find Jekyll is the right tool for the job. For me I probably would have been better off stiching a couple of node libraries together with Gulp or the like.

@nhoizey

This comment has been minimized.

Copy link
Owner

nhoizey commented Sep 22, 2017

Anyone still having this issue?

jekyll/jekyll#6344 might fix it.

@RRMoelker

This comment has been minimized.

Copy link

RRMoelker commented Sep 24, 2017

Yes, with Jekyll 3.6 the site is compiling without excluding the files in the Jekyll config. I do get errors during the build but those no longer halt the process.

So problem fixed,
though I'll keep my config so the build doesn't show any errors:

exclude:
  ...
  - _posts/**/*.jpg
  - _posts/**/*.png
  - _posts/**/*.gif
@nhoizey

This comment has been minimized.

Copy link
Owner

nhoizey commented Sep 24, 2017

Do you really get errors, or just informations? What is the color of the message?

If you exclude them, do it still copy them to the destination?

@DirtyF

This comment has been minimized.

Copy link
Contributor

DirtyF commented Sep 24, 2017

Here is a test of the current output in Jekyll 3.6.0 without the plugin and without excluding image files in the config:

~/code/jekyll/tests/default
❯ bundle exec jekyll -v
jekyll 3.6.0

~/code/jekyll/tests/default
❯ tree _posts
_posts
├── 2017
│   └── 07
│       ├── 2017-07-27-so-long-disqus-hello-webmentions.md
│       ├── dareboost-disqus-page-weight.png
│       └── disqus-guests.png
├── 2017-07-01-welcome-to-jekyll.markdown
├── 2017-08-25-new-post.markdown
└── 2017-08-26-another&post.markdown

2 directories, 6 files

~/code/jekyll/tests/default
❯ bundle exec jekyll build --trace
Configuration file: /Users/frank/code/jekyll/tests/default/_config.yml
            Source: /Users/frank/code/jekyll/tests/default
       Destination: /Users/frank/code/jekyll/tests/default/_site
 Incremental build: disabled. Enable with --incremental
      Generating...
                    done in 0.451 seconds.
 Auto-regeneration: disabled. Use --watch to enable.

WIth the plugin still no error messsages even with the --trace option, and the files are copied:

~/code/jekyll/tests/default
❯ tree _site/2017/07
_site/2017/07
├── 19
│   └── another&post.html
└── 27
    ├── dareboost-disqus-page-weight.png
    ├── disqus-guests.png
    └── so-long-disqus-hello-webmentions.html

Looks like you can close this issue 😃

@nhoizey

This comment has been minimized.

Copy link
Owner

nhoizey commented Sep 24, 2017

@DirtyF awesome, thanks for the detailed test!

@nhoizey nhoizey closed this Sep 24, 2017

@noaham

This comment has been minimized.

Copy link

noaham commented May 2, 2018

Hi. I am having the exact same issue. Jekyll version 3.7.3. My posts directory looks like

_posts/
    2018-05-01-first-post.md
    2018-05-02-post-in-directory/
        2018-05-02-post-in-directory.md
        one.jpg
        two.jpg
    2018-05-03-another-post.md

Running bundle exec jekyll build gives the error

Error: could not read file .../_posts/2018-05-02-post-in-directory/one.jpg: invalid byte sequence in UTF-8
Error: could not read file .../_posts/2018-05-02-post-in-directory/two.jpg: invalid byte sequence in UTF-8

However, when I rename the directory containing the image files, I no longer get an error. I.e. with the following directory structure, everything is fine.

_posts/
    2018-05-01-first-post.md
    post-in-directory/
        2018-05-02-post-in-directory.md
        one.jpg
        two.jpg
    2018-05-03-another-post.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment