No dotclear command in jekyll/commands/import.rb

[jrfern]

Sorry if this is plain silly, I was trying to use "jekyll import" (bundle-updated) to import my dotclear blog, as documented in https://import.jekyllrb.com/docs/dotclear/ and I get

jekyll 4.3.2 | Error: Whoops, we can't understand your command. jekyll 4.3.2 | Error: invalid option: --datafile jekyll 4.3.2 | Error: Run your command again with the --help switch to see available options. /home/jr/gems/gems/mercenary-0.4.0/lib/mercenary/program.rb:33:in go': invalid option: --datafile (OptionParser::InvalidOption)
`

"jekyll import --help" doesn't help. Tried to find out what's the matter, and I found a jekyll/commands/import.rb file that included all the importers but dotclear. Is it an oversight or has it been removed as obsolete?

[parkr]

Looks like a bug. We should have all the importers defined here.

Would you be up for a PR to add dotclear (and any other missing importers)? If so, mention me and I'll take a look. Otherwise, I'll try to fix this bug!

In the meantime, you could try:

$ ruby -r rubygems -e 'require "jekyll-import";
    JekyllImport::Importers::Dotclear.run({
      "datafile"      => "path/to/datafile",
      "mediafolder" => "path/to/mediafolder"
    })'

[parkr]

@ashmaroli There might be a way to convert the command's explicit hash into a dynamic one based on files in the importers/ subdirectory if you'd be interested in tackling that. It would reduce the work of creating new importers!

[ashmaroli]

It looks pretty simple to tackle; in theory, downcase klass.name of all JekyllImport::Importers::Importer subclasses..

I'll submit a PR by tomorrow.

[ashmaroli]

Hello @jrfern,
Could you please rerun the dotclear import command to see if this issue is reproducible?

[jrfern]

Issue reproduced. Remember that I use the bundle version, not the github one. Anyway I have git-cloned this repository, expecting some changes. No commit has been made, so both versions are identical. Just checked: same output - "invalid option: --datafile".

I also wrote that "jekyll import --help" is of no use. No mention of importers, no related parameters.

I keep on suspecting that at least a mention of dotclear should be added to jekyll/commands/import.rb.

[ashmaroli]

@jrfern There hasn't been any changes made to this repo since you last cloned. I just wanted to check if you entered the command name correctly.
Okay. So what was the exact incantation? Was it jekyll import dotclear ... or bundle exec jekyll import dotclear ...?

From what I realised, the code you and Parker pointed out isn't being used at all, so absence of dotclear in it is irrelevant.
What is output if you were to rerun your incantation without either of the option flags, i.e. just jekyll import dotclear or bundle exec jekyll import dotclear?

[ashmaroli]

@parkr From my tests, the code you have linked in your comment is not accessed at all when invoking an importer via CLI. This was confirmed at my end by invoking a non-existent importer:

bundle exec jekyll import jekyllbot

Expected to get the graceful error message, but got nothing ergo Jekyll::Commands::Import.process doesn't get invoked.
I'm handing this over back to you.

[jrfern]

@ashmaroli I tried the four options (with dotclear and with Dotclear). Here's the output

$ bundle exec jekyll import dotclear --datafile /home/MYUSER/.../2023-02-28-10-54-default-backup.txt --mediafolder /home/MYUSER/.../2023-02-28-default-media/img/
jekyll 4.3.2 | Error:  Whoops, we can't understand your command.
jekyll 4.3.2 | Error:  invalid option: --datafile
jekyll 4.3.2 | Error:  Run your command again with the --help switch to see available options.
/home/MYUSER/gems/gems/mercenary-0.4.0/lib/mercenary/program.rb:33:in `go': invalid option: --datafile (OptionParser::InvalidOption)
	from /home/MYUSER/gems/gems/mercenary-0.4.0/lib/mercenary.rb:21:in `program'
	from /home/MYUSER/gems/gems/jekyll-4.3.2/exe/jekyll:15:in `<top (required)>'
	from /home/MYUSER/gems/bin/jekyll:25:in `load'
	from /home/MYUSER/gems/bin/jekyll:25:in `<top (required)>'
	from /home/MYUSER/gems/gems/bundler-2.4.7/lib/bundler/cli/exec.rb:58:in `load'
	from /home/MYUSER/gems/gems/bundler-2.4.7/lib/bundler/cli/exec.rb:58:in `kernel_load'
	from /home/MYUSER/gems/gems/bundler-2.4.7/lib/bundler/cli/exec.rb:23:in `run'
	from /home/MYUSER/gems/gems/bundler-2.4.7/lib/bundler/cli.rb:492:in `exec'
	from /home/MYUSER/gems/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
	from /home/MYUSER/gems/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
	from /home/MYUSER/gems/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
	from /home/MYUSER/gems/gems/bundler-2.4.7/lib/bundler/cli.rb:34:in `dispatch'
	from /home/MYUSER/gems/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
	from /home/MYUSER/gems/gems/bundler-2.4.7/lib/bundler/cli.rb:28:in `start'
	from /home/MYUSER/gems/gems/bundler-2.4.7/exe/bundle:45:in `block in <top (required)>'
	from /home/MYUSER/gems/gems/bundler-2.4.7/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
	from /home/MYUSER/gems/gems/bundler-2.4.7/exe/bundle:33:in `<top (required)>'
	from /home/MYUSER/gems/bin/bundle:25:in `load'
	from /home/MYUSER/gems/bin/bundle:25:in `<main>'

$ jekyll import dotclear --datafile /home/MYUSER/.../2023-02-28-10-54-default-backup.txt --mediafolder /home/MYUSER/.../2023-02-28-default-media/img/
jekyll 4.3.2 | Error:  Whoops, we can't understand your command.
jekyll 4.3.2 | Error:  invalid option: --datafile
jekyll 4.3.2 | Error:  Run your command again with the --help switch to see available options.
/home/MYUSER/gems/gems/mercenary-0.4.0/lib/mercenary/program.rb:33:in `go': invalid option: --datafile (OptionParser::InvalidOption)
	from /home/MYUSER/gems/gems/mercenary-0.4.0/lib/mercenary.rb:21:in `program'
	from /home/MYUSER/gems/gems/jekyll-4.3.2/exe/jekyll:15:in `<top (required)>'
	from /home/MYUSER/gems/bin/jekyll:25:in `load'
	from /home/MYUSER/gems/bin/jekyll:25:in `<main>'

[ashmaroli]

@jrfern Okay now try running the following:

jekyll import
jekyll import tesla
jekyll import dotclear

[jrfern]

$ jekyll import
You must install the 'jekyll-import' gem version > 0 to use the 'jekyll import' command. 

$ jekyll import tesla
You must install the 'jekyll-import' gem version > 0 to use the 'jekyll import' command.

$ jekyll import dotclear
You must install the 'jekyll-import' gem version > 0 to use the 'jekyll import' command.

$ bundle exec jekyll import dotclear
You must install the 'jekyll-import' gem version > 0 to use the 'jekyll import' command. 

$ bundle exec jekyll import dotclear --datafile ... --mediafolder ...
... invalid option: --datafile

This was unexpected. I thought that maybe there was a conflict with a system gem, so checked as root:

# gem uninstall jekyll-import
Gem 'jekyll-import' is not installed

# gem list jekyll

*** LOCAL GEMS ***

No system jekyll gems. Now as my user

$ gem list jekyll

*** LOCAL GEMS ***

jekyll (4.3.2)
jekyll-feed (0.17.0)
jekyll-import (0.21.0)
jekyll-sass-converter (3.0.0)
jekyll-seo-tag (2.8.0)
jekyll-watch (2.2.1)

[ashmaroli]

I expected exactly these results once I got a better understanding of the source code @jrfern.

I however still don't understand how your bundle exec ... runs do not detect the jekyll-import gem when you insist that you installed the plugin via bundle install... Do you really have the plugin listed in your Gemfile?

Please delete your Gemfile.lock, run bundle show, and paste the output here.

[jrfern]

I however still don't understand how your bundle exec ... runs do not detect the jekyll-import gem when you insist that you installed the plugin via bundle install...

Did I? No, I followed the instructions in https://import.jekyllrb.com/docs/installation/, so I used

$ gem install jekyll-import

Anyway I run 'bundle update' time after time. Is that enough? Seems it isn't.

Do you really have the plugin listed in your Gemfile?

No - is that the cause of my problems? I did 'bundle install' now, no mention of jekyll-import.
Do I have to manually add it to the plugins list in the Gemfile? And if so, should it be documented in https://import.jekyllrb.com?
Would

group :jekyll_plugins do
  ...
  gem "jekyll-import", "~> 0.21"
end

be Ok?

[jrfern]

Well, after filling the Gemfile with

group :jekyll_plugins do
  ...
  gem "jekyll-import", "~> 0.21"
  gem "activesupport"
  gem "pp"
end

the errors are different:

can not load translations from /home/MYUSER/gems/gems/activesupport-7.0.4.2/lib/active_support/locale/en.yml: #<ArgumentError: wrong number of arguments (given 2, expected 1)> (I18n::InvalidLocaleData)
...
/home/MYUSER/gems/gems/safe_yaml-1.0.5/lib/safe_yaml.rb:37:in `unsafe_load_file': wrong number of arguments (given 2, expected 1)

Some progress, then. But the command line "jekyll import" is no longer so straightforward.

[ashmaroli]

Looks like there are multiple problems getting surfaced here, @jrfern.

The first problem is that the documentation has failed to impart the necessary information to novice users such as yourselves. Essentially, what every user has to know is about the impact of Bundler. Bundler and Gemfile work together. They cannot function independent of themselves. When a user runs bundle ... the Gemfile and only the Gemfile contents are taken into consideration.
You not listing gem "jekyll-import" in the Gemfile is the root cause of the error reported in this ticket. The "not listed" bit got overlooked when you mentioned having the plugin up-to-date via bundle update.

Unfortunately, there comes the second problem. If jekyll-import were actually not available in the "bundle", Jekyll CLI should have pointed it out but instead the error message complained about an unknown flag.

The third problem that surfaced here is the fact that the CLI doesn't really bail out when given invalid subcommand (e.g. bundle exec jekyll import tesla won't error out), only invalid flags... (e.g. bundle exec jekyll import tesla --musk will most likely error out).

The fourth problem (w.r.t ArgumentError) is quite new and will need additional testing which is the fifth problem. The test-suite for this project is seriously lacking and therefore unreliable. 😭 😭

[parkr]

I'd vote here to use a new jekyll-import executable instead of the subcommand to eliminate the boot dependency on Jekyll.

[ashmaroli]

I'd vote here to use a new jekyll-import executable...

I'd vote against that idea (for now atleast). A new executable with an entirely different approach is expensive. We need a clear idea / design; change all existing (bundled) importers to play compatibly with new executable..

Right now, the major issue at hand is lack of a proper test-suite. The Dotclear importer among many others, isn't continuously tested. I have no idea regarding the schematics of a Dotclear datafile which is a mandatory requirement for the importer. So I can't test things at my end and have to rely on feedback from non-developer users who are not familiar with the code under the hood. 🤷

[parkr]

I don't think it would be expensive. I can try to spend some time on it this weekend. It would still be a mercenary command but would work out of the box without Jekyll.

In the meantime our docs don't work right? If so we should consider fixing them.

[ashmaroli]

@parkr can I request you to work on getting the test-suite reliable? I would really love to have mock schema / export-file for as many importers as practically possible.
Again, the issue with our current CLI is fixable / manageable, but issues with importers themselves are more serious.

[jrfern]

feedback from non-developer users who are not familiar with the code under the hood

That's me :-). As an external observer and a non-developer I would consider that the software should keep to its promises, or the documentation should be changed. I also understand that there are always too many things to do and too few people to do them, and that priorities must be decided.

As I got to the fourth problem level (@ashmaroli) here --no longer such a newbie-- there are two tasks now: solve it and test-suite identify it. The first one is logically prior to the second one, isn't it? How can I help identify what's happening? Dotclear's backup.txt is a utf-8 file, and I suppose that everything is utf-8 by default, so why the InvalidLocaleData error? Any configuration line I should have added somewhere?

[jrfern]

$ bundle exec jekyll import --help
jekyll import -- Import your old blog to Jekyll

Usage:

  jekyll import <platform> [options]

Options:

        -s, --source [DIR]  Source directory (defaults to ./)
        -d, --destination [DIR]  Destination directory (defaults to ./_site)
            --safe         Safe mode (defaults to false)
        -p, --plugins PLUGINS_DIR1[,PLUGINS_DIR2[,...]]  Plugins directory (defaults to ./_plugins)
            --layouts DIR  Layouts directory (defaults to ./_layouts)
            --profile      Generate a Liquid rendering profile
        -h, --help         Show this message
        -v, --version      Print the name and version
        -t, --trace        Show the full backtrace when an error occurs

Subcommands:
  rss                   
...

This keeps puzzling me. I created the structure with "bundle exec jekyll new ...". The help output mentions a plugins subdirectory, should I create it manually? _config.yml mentions jekyll-feed.

Also I don't know how to cope with the --source option, that contradicts the line "jekyll import dotclear --datafile PATH --mediafolder PATH".

[ashmaroli]

How can I help identify what's happening?

In theory @jrfern, you could alter the Dotclear backup file to not contain any identifying data, commit the file into a git repository and push that up to GitHub and give us the URL so that we can clone that repo.
I won't be able to personally look into the above repository contents until Sunday, so there's no hurry. Take your time to convert the backup file into a generic schematic file.
What do I mean by a generic schematic file?
Well, all backup files contain a pattern of data that a given software parses into information used to "reconstruct" the source. Our Dotclear importer is one such software.
Say, you have 10 posts in your Dotclear blog. That means you'll have at least 10 sections of data that follow a certain pattern in the backup file. For test purposes, just 2 such data blocks would suffice. Create a copy of the backup file, replace any blog-specific data with arbitrary stuff. e.g., "John Doe" for name, "Lorem ipsum dolor sit ..." for post content, arbitrary URLs for any asset files, etc.

This keeps puzzling me. I created the structure with "bundle exec jekyll new ...". The help output mentions a plugins subdirectory...

All of those options are actually injected by Jekyll and have no relevance to jekyll-import. I guess this is why @parkr suggested introducing an entirely new CLI executable for jekyll-import instead of being an extension of the jekyll CLI executable. (Basically, bundle exec jekyll-import dotclear ... instead of what we currently recommend; note the hyphenation.)

[jrfern]

https://bpa.st/3J6BE
It will be available for a week.