Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a new Metanorma Document #35

Merged
merged 16 commits into from Apr 15, 2019

Conversation

Projects
None yet
3 participants
@abunashir
Copy link
Member

commented Mar 27, 2019

This commit adds a very simple and basic interface to create a new metanorma document, it currently supports CSD document but very soon we will add all of the other supporting document.

The updates our existing executable exe/metanorma with support to switch to the old version (for now). This new changes will allow us to create a new document, and the way we can do it now is to use the following command.

# check exe/metanorma help for full options
exe/metanorma my-csd-document -t csd -d standard

Fair warning, this is still in very early stage to support our use case, but we will soon clean up this and make it as friendly as possible.

@ronaldtse

This comment has been minimized.

Copy link
Contributor

commented Mar 29, 2019

Few changes needed (feel free to merge this first) following #36.

exe/metanorma-new new -t csd -d standard my-csd-document

Now creates this structure:

├── my-csd-document
│   ├── Gemfile
│   ├── Makefile
│   ├── Makefile.win
│   ├── appveyor.yml
│   ├── deploy.sh
│   └── standard
│       ├── README.adoc
│       ├── cc-document.adoc
│       └── sections
│           ├── 01-scope.adoc
│           ├── 02-norm-refs.adoc
│           ├── 03-termsdef.adoc
│           ├── 04-content.adoc
│           ├── 0a-foreword.adoc
│           ├── 0b-intro.adoc
│           ├── a1-annex-1.adoc
│           └── zz-references.adoc

I'd like this instead:

├── my-csd-document
│   ├── Gemfile
│   ├── Makefile
│   ├── Makefile.win
│   ├── README.adoc
│   ├── appveyor.yml
│   ├── cc-document.adoc
│   ├── deploy.sh
│   └── sections
│       ├── 01-scope.adoc
│       ├── 02-norm-refs.adoc
│       ├── 03-termsdef.adoc
│       ├── 04-content.adoc
│       ├── 0a-foreword.adoc
│       ├── 0b-intro.adoc
│       ├── a1-annex-1.adoc
│       └── zz-references.adoc

In addition, some people like a "flattened" document, i.e. the includes should be flattened. Or, we can provide another set of templates that are "flattened".

We will want to use a command line argument for this because it is now very clear that these two audiences ("hierarchical!", "single file!") are very different.

e.g.

├── my-csd-document
│   ├── Gemfile
│   ├── Makefile
│   ├── Makefile.win
│   ├── README.adoc
│   ├── appveyor.yml
│   ├── cc-document.adoc <<< All separate sections are subsumed into here
│   ├── deploy.sh

We probably want a command line argument to allow user to pull from a their own custom "template repository". e.g.

exe/metanorma-new new -t foo -d bar --template-repository=https://github.com/foo/metanorma-foo-templates my-special-document

In generator.rb:
Screen Shot 2019-03-29 at 12 13 13 PM

Maybe we don't even need to hard code this? e.g. Homebrew's brew tap [username]/[reponame] assumes the formula (git) repo is already at https://github.com/[username]/homebrew-[reponame]`.

@ronaldtse

This comment has been minimized.

Copy link
Contributor

commented Apr 1, 2019

@abunashir please also add the OGC templates: https://github.com/metanorma/mn-templates-ogc (flavor = :ogc).

Part of me thinks that the definition of templates_sources should be located in each of the gems themselves and loaded through a Template Registry. Thoughts?

@ronaldtse ronaldtse referenced this pull request Apr 3, 2019

Closed

Command `metanorma new` #13

@ronaldtse

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2019

We're going to change the command line to:

metanorma new -t {type} -d {template-name} [my-new-document-name]
# or
metanorma new {type}/{template-name} [my-new-document-name]

Which will create a directory at {my-new-document-name}.

@abunashir

This comment has been minimized.

Copy link
Member Author

commented Apr 3, 2019

Hi @ronaldtse, there are couple of concerns, let's me try to add those in your comments, and then we can take it from there.

Few changes needed (feel free to merge this first) following #36.

exe/metanorma-new new -t csd -d standard my-csd-document

Now creates this structure:

├── my-csd-document
│   ├── Gemfile
│   ├── Makefile
│   ├── Makefile.win
│   ├── appveyor.yml
│   ├── deploy.sh
│   └── standard
│       ├── README.adoc
│       ├── cc-document.adoc
│       └── sections
│           ├── 01-scope.adoc
│           ├── 02-norm-refs.adoc
│           ├── 03-termsdef.adoc
│           ├── 04-content.adoc
│           ├── 0a-foreword.adoc
│           ├── 0b-intro.adoc
│           ├── a1-annex-1.adoc
│           └── zz-references.adoc

I'd like this instead:

├── my-csd-document
│   ├── Gemfile
│   ├── Makefile
│   ├── Makefile.win
│   ├── README.adoc
│   ├── appveyor.yml
│   ├── cc-document.adoc
│   ├── deploy.sh
│   └── sections
│       ├── 01-scope.adoc
│       ├── 02-norm-refs.adoc
│       ├── 03-termsdef.adoc
│       ├── 04-content.adoc
│       ├── 0a-foreword.adoc
│       ├── 0b-intro.adoc
│       ├── a1-annex-1.adoc
│       └── zz-references.adoc

This looks good to me for single type of document (eg: standard/collection), is there any chance a user might want to add multiple type of document in one project, eg: standard and admin combined, then what be structure for this flatten hiaracy?

In addition, some people like a "flattened" document, i.e. the includes should be flattened. Or, we can provide another set of templates that are "flattened".

We will want to use a command line argument for this because it is now very clear that these two audiences ("hierarchical!", "single file!") are very different.

e.g.

├── my-csd-document
│   ├── Gemfile
│   ├── Makefile
│   ├── Makefile.win
│   ├── README.adoc
│   ├── appveyor.yml
│   ├── cc-document.adoc <<< All separate sections are subsumed into here
│   ├── deploy.sh

Yap, I agree on this one, we can add a command line argument later and thats how we can allow both type of user the flexibility to use different structure.

We probably want a command line argument to allow user to pull from a their own custom "template repository". e.g.

exe/metanorma-new new -t foo -d bar --template-repository=https://github.com/foo/metanorma-foo-templates my-special-document

In generator.rb:
Screen Shot 2019-03-29 at 12 13 13 PM

Maybe we don't even need to hard code this? e.g. Homebrew's brew tap [username]/[reponame] assumes the formula (git) repo is already at https://github.com/[username]/homebrew-[reponame]`.

Yap, that's my whole idea to allow user to provide templates as an argument, but for our tempaltes I think we still should have one based hard coded sources as a fall back, so in any case it would work and we also make sure the structrue and everythig is right otherwise there would be lot more reason that could beak our things.

Please have a look and let me know your thoughts on those, thanks!

@abunashir

This comment has been minimized.

Copy link
Member Author

commented Apr 3, 2019

@abunashir please also add the OGC templates: https://github.com/metanorma/mn-templates-ogc (flavor = :ogc).

I assume the above once is done, right?

Part of me thinks that the definition of templates_sources should be located in each of the gems themselves and loaded through a Template Registry. Thoughts?

Since, it's only a link to template repos, so I feel like it might be much more straight forward to have the base list of templates directly here, otherwise we just add additional depencey to another repo, don't you think?

@opoudjis

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2019

Btw, I'm doing Metanorma gem releases now, but given this is WIP, I'm not releasing metanorma-cli

@ronaldtse

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2019

This looks good to me for single type of document (eg: standard/collection), is there any chance a user might want to add multiple type of document in one project, eg: standard and admin combined, then what be structure for this flatten hiaracy?

No, a user will only want a single template for now. It will be very rare for people to want different components from different document types, but if they really wanted to, they can download the templates and manually do so 😉 (we should add this in the documentation).

Yap, that's my whole idea to allow user to provide templates as an argument, but for our tempaltes I think we still should have one based hard coded sources as a fall back, so in any case it would work and we also make sure the structrue and everythig is right otherwise there would be lot more reason that could beak our things.

Agree for now.

I assume the above once is done, right? ("OGC")

Yes!

Thanks!

@ronaldtse

This comment has been minimized.

Copy link
Contributor

commented Apr 4, 2019

@abunashir also would it be better to use https://github.com/ruby-git/ruby-git rather than calling the git executable directly?

abunashir and others added some commits Mar 27, 2019

Update gitignroe file & add binstub
* Remove Gemfile.lock from the repository
* Do not commit the remote rubocop in gith
* Add binstub to run rspec in current scope
Add support to create a new document
This commit adds a very simple and basic interface to create a new
metanorma document, it currently supports csd document but very soon
we will add all of the other supported document.

The adds a temporary new executable `exe/metanorma-new` and this
will allow us to create a new document, and the way we can do it
now is to use the following command.

```sh
exe/metanorma-new new -t csd -d standard my-csd-document
```

Fair warning, this is still in very early stage to support our use
case, but we will soon clean up this and make it as friendly as
possible.
Adopt thor gem for the metanorma cli
The current version of metanorma cli is more likely a single
ruby file, and we are doing all the option parsing and others
manually. This also makes it bit harder to extend, and also
forces us to re-write some common behavior by ourself.

This commit changes this, and adopt the most commonly used
library `thor` for this purpose, it also migrate all existing
behavior to this new one

Since `metanorma` is being used in production, and this new
version is still in early stage, so instead of replacing it
directly, we are doing a soft release. If you wanna give it
a shot then you can do so by setting up an environment var
`METANORMA_DEV_MODE` to `true`.

@abunashir abunashir force-pushed the metanorma-new branch from f347969 to 08eeee3 Apr 7, 2019

Remove doctype from generated document
Currently the generated document contains a folder structure by
the doctype and then have it's item nested in it, but we actually
don't want it but to keep a flatten hierarchy.

This commit add that functionality, and instead of nesting the
files under doctype folder, it will now directly add those to
the root, and keep the other structure as it is.
@abunashir

This comment has been minimized.

Copy link
Member Author

commented Apr 7, 2019

In addition, some people like a "flattened" document, i.e. the includes should be flattened. Or, we can provide another set of templates that are "flattened".

We will want to use a command line argument for this because it is now very clear that these two audiences ("hierarchical!", "single file!") are very different.

@ronaldtse: My thought in this one, if we can have some reliable lirabary that could flatten the include directive without messing up the format or anything that would be perfect.

Otherwise I would say maybe it might be safe have another template file like csd-flatten.adoc in the template directory and we use that one when user pass the flatten option, What do you think? Are you aware of any such libararay?

cc: @opoudjis, @ribose-jeffreylau

@ronaldtse

This comment has been minimized.

Copy link
Contributor

commented Apr 8, 2019

@ronaldtse: My thought in this one, if we can have some reliable lirabary that could flatten the include directive without messing up the format or anything that would be perfect.

I think what @opoudjis thinks is that we can do it.

Otherwise I would say maybe it might be safe have another template file like csd-flatten.adoc in the template directory and we use that one when user pass the flatten option, What do you think?

However it might be easiest to have a separate "flattened" template. Or, when we publish the templates repository, we automatically generate the flattened template.

How should we name them?

  • Hierarchical: {template}
  • Flattened: {template}-flat}

?

@ronaldtse

This comment has been minimized.

Copy link
Contributor

commented Apr 8, 2019

@abunashir is this ready to merge?

@abunashir

This comment has been minimized.

Copy link
Member Author

commented Apr 9, 2019

For the Hierarchical one, we can keep as it is right now, and add another template-flat file that we will use when the user pass the flattern option.

How should we name them?

  • Hierarchical: {template}
  • Flattened: {template}-flat}
@abunashir

This comment has been minimized.

Copy link
Member Author

commented Apr 9, 2019

@abunashir is this ready to merge?

There are couple of things I would prefer to add frist and do some cleanup, but you can also merge it if you are planning to use straight away, I can then work on top of that.

Btw, to use the latest version please set up METANORMA_DEV_MODE environment variable first, Thanks!

@ronaldtse

This comment has been minimized.

Copy link
Contributor

commented Apr 9, 2019

For the Hierarchical one, we can keep as it is right now, and add another template-flat file that we will use when the user pass the flattern option.

OK

Btw, to use the latest version please set up METANORMA_DEV_MODE environment variable first, Thanks!

Can we not require usage of METANORMA_DEV_MODE? We want it to just work 😉

@abunashir

This comment has been minimized.

Copy link
Member Author

commented Apr 9, 2019

Ideally, I think it might be safe to keep the old version active for a while since it is being used it multiple libraries and this one has not been tested well yet, but sure we can do that. Just let me know, so i can remove that line from there :)

Can we not require usage of METANORMA_DEV_MODE? We want it to just work 😉

@ronaldtse

This comment has been minimized.

Copy link
Contributor

commented Apr 10, 2019

@abunashir the specs are now properly done, but the failures are real functionality gaps / bugs. Could you please help fix them? Thanks!

@abunashir

This comment has been minimized.

Copy link
Member Author

commented Apr 11, 2019

Hi @ronaldtse, fixing the specs and also migrating to Aruba was casing some issue I believe, so I've extracted those here: #41, and once this one is done then let's revisit that again and we can do the migration.

@ronaldtse

This comment has been minimized.

Copy link
Contributor

commented Apr 11, 2019

@abunashir the issues are not caused by Aruba but are real issues with the CLI. Can you please help fix them?

  1) Metanorma processes metanorma options inside Asciidoc creates proper output
     Failure/Error: expect(exist?("test.html")).to be true
       expected true
            got false

  2) Metanorma file with blank header processes an asciidoc ISO document creates proper output
     Got 3 failures from failure aggregation block.
...

  3) Metanorma file with blank header processes all extensions of an asciidoc ISO document creates proper output
...

  4) Metanorma file with configured header extracts isodoc options from Asciidoc file creates proper output
...

These are all caused by the CLI's handling of the -x {ext,ext...} option.

  5) Metanorma file with configured header wraps HTML output wraps with directories
     Got 3 failures from failure aggregation block.
     # ./spec/metanorma_spec.rb:127:in `block (4 levels) in <top (required)>'
     # ./vendor/bundle/ruby/2.5.0/gems/rspec-command-1.0.3/lib/rspec_command.rb:47:in `block (2 levels) in <module:RSpecCommand>'
     # ./vendor/bundle/ruby/2.5.0/gems/rspec-command-1.0.3/lib/rspec_command.rb:45:in `block in <module:RSpecCommand>'
     # ./vendor/bundle/ruby/2.5.0/gems/aruba-0.14.9/lib/aruba/rspec.rb:22:in `block (2 levels) in <top (required)>'
     5.1) Failure/Error: expect(last_command_started).to be_successfully_executed
            expected "metanorma -w -t iso test.adoc" to be successfully executed
          # ./spec/metanorma_spec.rb:128:in `block (5 levels) in <top (required)>'
     5.2) Failure/Error: expect(exist?("test/test.html")).to be true
            expected true
                 got false
          # ./spec/metanorma_spec.rb:129:in `block (5 levels) in <top (required)>'
     5.3) Failure/Error: expect(exist?("test.alt/test.alt.html")).to be true
            expected true
                 got false
          # ./spec/metanorma_spec.rb:130:in `block (5 levels) in <top (required)>'

This is caused by the -w "wrapping" option not working.

  8) Metanorma file with configured header exports bibdata generates Relaton file
     Failure/Error: expect(last_command_started).to be_successfully_executed
       expected "metanorma -R testrelaton.xml -t iso test.adoc" to be successfully executed
     # ./spec/metanorma_spec.rb:183:in `block (5 levels) in <top (required)>'
     # ./spec/metanorma_spec.rb:182:in `block (4 levels) in <top (required)>'
     # ./vendor/bundle/ruby/2.5.0/gems/rspec-command-1.0.3/lib/rspec_command.rb:47:in `block (2 levels) in <module:RSpecCommand>'
     # ./vendor/bundle/ruby/2.5.0/gems/rspec-command-1.0.3/lib/rspec_command.rb:45:in `block in <module:RSpecCommand>'
     # ./vendor/bundle/ruby/2.5.0/gems/aruba-0.14.9/lib/aruba/rspec.rb:22:in `block (2 levels) in <top (required)>'

This is caused by the -R {file} option not working.

  9) Metanorma file with configured header exports assets extracts assets in specified folder
     Got 2 failures from failure aggregation block.
     # ./spec/metanorma_spec.rb:228:in `block (4 levels) in <top (required)>'
     # ./vendor/bundle/ruby/2.5.0/gems/rspec-command-1.0.3/lib/rspec_command.rb:47:in `block (2 levels) in <module:RSpecCommand>'
     # ./vendor/bundle/ruby/2.5.0/gems/rspec-command-1.0.3/lib/rspec_command.rb:45:in `block in <module:RSpecCommand>'
     # ./vendor/bundle/ruby/2.5.0/gems/aruba-0.14.9/lib/aruba/rspec.rb:22:in `block (2 levels) in <top (required)>'
     9.1) Failure/Error: expect(exist?("extract/sourcecode/sourcecode-0000.txt")).to be true
            expected true
                 got false
          # ./spec/metanorma_spec.rb:234:in `block (5 levels) in <top (required)>'
     9.2) Failure/Error: expect("extract/sourcecode/sourcecode-0000.txt").to have_file_content EXPECTED_CONTENT
            expected "extract/sourcecode/sourcecode-0000.txt" to have file content: "def ruby(x)\n  if x < 0 && x > 1\n    return\n  end\nend\n"
          # ./spec/metanorma_spec.rb:235:in `block (5 levels) in <top (required)>'

This is caused by the extract option not working.

  10) Metanorma warns when no file provided should have output: /Need to specify a file to process/
      Failure/Error: it { expect(last_command_started).to have_output(/Need to specify a file to process/) }
        expected "ERROR: \"metanorma compile\" was called with no arguments\nUsage: \"metanorma compile FILENAME\"" to have output: /Need to specify a file to process/
        Diff:
        @@ -1,2 +1,3 @@
        -/Need to specify a file to process/
        +ERROR: "metanorma compile" was called with no arguments
        +Usage: "metanorma compile FILENAME"
      # ./spec/metanorma_spec.rb:297:in `block (3 levels) in <top (required)>'
      # ./vendor/bundle/ruby/2.5.0/gems/rspec-command-1.0.3/lib/rspec_command.rb:47:in `block (2 levels) in <module:RSpecCommand>'
      # ./vendor/bundle/ruby/2.5.0/gems/rspec-command-1.0.3/lib/rspec_command.rb:45:in `block in <module:RSpecCommand>'
      # ./vendor/bundle/ruby/2.5.0/gems/aruba-0.14.9/lib/aruba/rspec.rb:22:in `block (2 levels) in <top (required)>'

This is caused by a difference in error message expectation (new CLI uses default Thor message).

@abunashir abunashir force-pushed the metanorma-new branch from 3be82b9 to fe53db7 Apr 11, 2019

@abunashir

This comment has been minimized.

Copy link
Member Author

commented Apr 11, 2019

Yap, there are quite some issues on the CLI arguments/interface, I'm working on fixing those first

Fix all migration related issues
During the migration of the new version, we've added some default
value to some of the options and that's what was breaking some of
the existing functionality.

We were also passing the options keys as string, but previously
we were passing it as symbols, and that was also creating some
issue in this new version.

This commit fixes all of these issues, and it tries to bring it
as close as possible toward the existing version.

@abunashir abunashir force-pushed the metanorma-new branch from fe53db7 to 9ed9c69 Apr 11, 2019

@abunashir

This comment has been minimized.

Copy link
Member Author

commented Apr 11, 2019

Hi @ronaldtse, this latest changes should fix all of the existing functionality, could you please give it a try when the test passes?

@abunashir abunashir force-pushed the metanorma-new branch from 65c12a1 to 213090a Apr 11, 2019

@ronaldtse

This comment has been minimized.

Copy link
Contributor

commented Apr 12, 2019

Thanks @abunashir ! There's a single spec still failing on AppVeyor:

  1) Metanorma processes metanorma options inside Asciidoc
530     Failure/Error: expect(File.exist?("test.xml")).to be true
531
532       expected true
533            got false
534     # ./spec/metanorma_spec.rb:8:in `block (2 levels) in <top (required)>'
535
536Finished in 32.59 seconds (files took 0.69932 seconds to load)
53720 examples, 1 failure
538
539Failed examples:
540
541

Everything else is working -- it's strange that the "first spec" has this basic failure 😉 Could you help take a look at this? Thanks!

@abunashir abunashir force-pushed the metanorma-new branch 2 times, most recently from 59c2bba to 1339006 Apr 12, 2019

Fix basic test failure for appveyor CI
In this test, we were using the executable directly, and most likely
it was not exporting the file properly, so in this commit we changed
this to the executable, so it should write / export files for test
in windows properly.

@abunashir abunashir force-pushed the metanorma-new branch from cf593be to b1e1505 Apr 12, 2019

@abunashir

This comment has been minimized.

Copy link
Member Author

commented Apr 12, 2019

Hi @ronaldtse, all the issue should be fixed now, could you please double check and merge it when you are happy with these changes :) Thanks!

@opoudjis

This comment has been minimized.

Copy link
Contributor

commented Apr 14, 2019

This has just missed the window for gem releases, but I can do a release out of sequence if you let me know; I'm already going to have to do one for ietfbib.

Would rather @ronaldtse do the review, since he's been involved in the discussions.

Show resolved Hide resolved .travis.yml
Show resolved Hide resolved spec/acceptance/new_spec.rb
Show resolved Hide resolved spec/metanorma/cli/generator_spec.rb
Show resolved Hide resolved spec/metanorma_spec.rb
@ronaldtse

This comment has been minimized.

Copy link
Contributor

commented Apr 15, 2019

I'm going to first merge this and make a new issue for the requested changes, since they are only relevant to testing. Thanks @abunashir !

@ronaldtse ronaldtse merged commit 54d4f7d into master Apr 15, 2019

4 checks passed

continuous-integration/appveyor/branch AppVeyor build succeeded
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.