Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename annotation to marker #24

Merged
merged 27 commits into from
Jul 27, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
7bbe553
Replace annotation with marker
rytswd Jul 22, 2021
3a58c60
Adjust marker note
rytswd Jul 22, 2021
393dd4e
Rename filenames
rytswd Jul 22, 2021
f17fd43
Rename annotation to marker in code
rytswd Jul 22, 2021
1e80812
Refactor redundant function names
rytswd Jul 22, 2021
c6f1774
Rename to Exporter Marker
rytswd Jul 22, 2021
02bcc87
Adjust testdata with marker
rytswd Jul 22, 2021
7bc72f1
Rename function back
rytswd Jul 22, 2021
0440a36
Introduce new package for marker and regex
rytswd Jul 24, 2021
70bbacf
Rename function
rytswd Jul 26, 2021
a7d2448
Fix test cases
rytswd Jul 26, 2021
f0213fe
Add clarification comment
rytswd Jul 26, 2021
690bb21
Refactor use of regexp with map solution
rytswd Jul 26, 2021
256e24c
Add another test case, minor wording change
rytswd Jul 27, 2021
f28f509
Comment correction
rytswd Jul 27, 2021
b9042e9
Adjust wording
rytswd Jul 27, 2021
3a7c6f1
Remove redundant files
rytswd Jul 27, 2021
f319fe4
Ensure test coverage
rytswd Jul 27, 2021
a81cc51
Consolidate use of marker to package
rytswd Jul 27, 2021
ac677b1
Add test case for non-markdown and non-yaml
rytswd Jul 27, 2021
a959702
Add more test cases
rytswd Jul 27, 2021
1943c81
Add print tests
rytswd Jul 27, 2021
71a8ab2
Refactor file structure
rytswd Jul 27, 2021
87e735c
Duplicate testdata to type based directories
rytswd Jul 27, 2021
43abc70
Update testdata usage
rytswd Jul 27, 2021
9dad43b
Correct file references
rytswd Jul 27, 2021
0c0ce6e
Move files around and correct references
rytswd Jul 27, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 22 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,41 +35,41 @@ $ go get github.com/upsidr/importer/cmd/importer@v0.0.1-rc2

<!-- == imptr: getting-started-example-short / begin from: ./docs/getting-started/examples.md#[simple] == -->

Let's see what Importer does with the file in this repository [`./testdata/simple-before.md`](https://raw.githubusercontent.com/upsidr/importer/main/testdata/simple-before.md).
Let's see what Importer does with the file in this repository [`./testdata/markdown/simple-before.md`](https://raw.githubusercontent.com/upsidr/importer/main/testdata/simple-before.md).

```markdown
# Simple Markdown Test

<!-- == imptr: lorem / begin from: ../docs/template/_lorem.md#5~12 == -->
<!-- == imptr: lorem / begin from: ../../docs/template/_lorem.md#5~12 == -->

Any content here will be removed by Importer.

<!-- == imptr: lorem / end == -->

Content after annotation is left untouched.
Content after marker is left untouched.
```

When you run `importer purge ./testdata/simple-before.md`:
When you run `importer purge ./testdata/markdown/simple-before.md`:

```bash
$ importer purge ./testdata/simple-before.md
$ cat ./testdata/simple-before.md
$ importer purge ./testdata/markdown/simple-before.md
$ cat ./testdata/markdown/simple-before.md
# Simple Markdown Test

<!-- == imptr: lorem / begin from: ../docs/template/_lorem.md#5~12 == -->
<!-- == imptr: lorem / begin from: ../../docs/template/_lorem.md#5~12 == -->
<!-- == imptr: lorem / end == -->

Content after annotation is left untouched.
Content after marker is left untouched.
```

When you run `importer generate ./testdata/simple-before.md`:
When you run `importer generate ./testdata/markdown/simple-before.md`:

```bash
$ importer generate ./testdata/simple-before.md
$ cat ./testdata/simple-before.md
$ importer generate ./testdata/markdown/simple-before.md
$ cat ./testdata/markdown/simple-before.md
# Simple Markdown Test

<!-- == imptr: lorem / begin from: ../docs/template/_lorem.md#5~12 == -->
<!-- == imptr: lorem / begin from: ../../docs/template/_lorem.md#5~12 == -->
"Lorem ipsum dolor sit amet,
consectetur adipiscing elit,
sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Expand All @@ -80,7 +80,7 @@ Excepteur sint occaecat cupidatat non proident,
sunt in culpa qui officia deserunt mollit anim id est laborum."
<!-- == imptr: lorem / end == -->

Content after annotation is left untouched.
Content after marker is left untouched.
```

<!-- == imptr: getting-started-example-short / end == -->
Expand Down Expand Up @@ -122,30 +122,30 @@ You can find actually running CI setup in [`.github/workflows/importer-markdown-

<!-- == imptr: getting-started-github-action / end == -->

## 🖋 Annotations and Markers
## 🖋 Markers

### Importer Annotation
### Importer Marker

<!-- == imptr: basic-annotation / begin from: ./docs/getting-started/annotations.md#[basic-annotation] == -->
<!-- == imptr: basic-marker / begin from: ./docs/getting-started/markers.md#[basic-marker] == -->

An annotation is a simple comment with special syntax, and thus is slightly different depending on file used.
A marker is a simple comment with special syntax, and thus is slightly different depending on file used.

The below is a simple example for **Markdown**.

```markdown
<!-- == imptr: getting-started-install / begin from: ./docs/getting-started/install.md#[homebrew-install] == -->
```

![Annotation explained][annotation-explanation]
![Marker explained][marker-explanation]

[annotation-explanation]: /assets/images/annotation-explanation.png "Annotation Explanation"
[marker-explanation]: /assets/images/marker-explanation.png "Marker Explanation"

And there has to be a matching "end" annotation. This is much simpler, as options are all defined in the "begin" annotation.
And there has to be a matching "end" marker. This is much simpler, as options are all defined in the "begin" marker.

```markdown
<!-- == imptr: getting-started-install / end == -->
```

<!-- == imptr: basic-annotation / end == -->
<!-- == imptr: basic-marker / end == -->

You can find more about the Importer Annotation [here](./docs/getting-started/annotations.md).
You can find more about the Importer Marker [here](./docs/getting-started/markers.md).
16 changes: 8 additions & 8 deletions docs/details/details.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,11 @@ That's why Importer was created. You can stick to simple file format, but when i

## Implementation Details

Importer is a simple regular expression file reader. It looks for special Importer Annotation comment, which has no meaning in that given language, but Importer can parse such comment and wires up other file content. [You can find more about Importer Annotation here.](../getting-started/annotatinos.md)
Importer is a simple regular expression file reader. It looks for special Importer Marker comment, which has no meaning in that given language, but Importer can parse such comment and wires up other file content. [You can find more about Importer Marker here.](../getting-started/annotatinos.md)

In language like Markdown and YAML, a single file is made to be as is, meaning you cannot import other files. This makes them really simple and easy to get started, but when you try to do something a bit more involved, it becomes difficult to maintain very quickly.

Because Importer tries to be "dumb", it doesn't actually know much about the given file syntax. Importer looks for Importer Annotation comments, parses them, and generates the updated version of that file, with specified lines imported into it.
Because Importer tries to be "dumb", it doesn't actually know much about the given file syntax. Importer looks for Importer Marker comments, parses them, and generates the updated version of that file, with specified lines imported into it.

Because the goal of Importer is very simple, the implementation is based on simple regular expressions. It is not made to be performant, nor capable of handling complex scenarios. But it works for most cases, such as Markdown and YAML. Other file typse may benefit from this approach. If there is any other file types that could benefit from this, we will look to expand our support in the future.

Expand All @@ -43,22 +43,22 @@ Importer is planning to support YAML files in the near future.

Currently, `importer generate` takes in a file argument, and updates the file content in place.

Instead, we are aiming to provide `importer update` to provide the same feature, while `importer generate` to output the result to stdout. This allows having a separate file that contains Importer Annotations, and a generated file as a separate file.
Instead, we are aiming to provide `importer update` to provide the same feature, while `importer generate` to output the result to stdout. This allows having a separate file that contains Importer Markers, and a generated file as a separate file.

### Add `graph` command

Currently, Importer only looks at the provided argument and its Import Target Files. When the Target File contains another Importer Annotation, it would be better to update the Target File content first.
Currently, Importer only looks at the provided argument and its Import Target Files. When the Target File contains another Importer Marker, it would be better to update the Target File content first.
We will need much better processing than simple regex handling, and abstract syntax tree needs to be created for this command. Also, this command needs to ensure there is no cyclic dependencies in the Importer definitions.

### Support line brak in Importer Annotation and Export Marker
### Support line brak in Importer Marker and Exporter Marker

Currently Importer Annotation and Export Marker have to be a single line input. If you have a line break in them, it will be ignored. This is because how it's currently implemented, and fixing this would require a proper AST setup when parsing a file.
Currently Importer Marker and Exporter Marker have to be a single line input. If you have a line break in them, it will be ignored. This is because how it's currently implemented, and fixing this would require a proper AST setup when parsing a file.

### Add special annotations `ignore` to skip Importer run
### Add special markers `ignore` to skip Importer run

When having an automation such as `find . -name '*.md' -exec importer generate {} \;`, you may want to skip some files.

This shouldn't skip Export Marker handling, though.
This shouldn't skip Exporter Marker handling, though.

### Add `diff` command

Expand Down
12 changes: 6 additions & 6 deletions docs/details/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,22 +16,22 @@ Importer is planning to support YAML files in the near future.

Currently, `importer generate` takes in a file argument, and updates the file content in place.

Instead, we are aiming to provide `importer update` to provide the same feature, while `importer generate` to output the result to stdout. This allows having a separate file that contains Importer Annotations, and a generated file as a separate file.
Instead, we are aiming to provide `importer update` to provide the same feature, while `importer generate` to output the result to stdout. This allows having a separate file that contains Importer Markers, and a generated file as a separate file.

### Add `graph` command

Currently, Importer only looks at the provided argument and its Import Target Files. When the Target File contains another Importer Annotation, it would be better to update the Target File content first.
Currently, Importer only looks at the provided argument and its Import Target Files. When the Target File contains another Importer Marker, it would be better to update the Target File content first.
We will need much better processing than simple regex handling, and abstract syntax tree needs to be created for this command. Also, this command needs to ensure there is no cyclic dependencies in the Importer definitions.

### Support line brak in Importer Annotation and Export Marker
### Support line brak in Importer Marker and Exporter Marker

Currently Importer Annotation and Export Marker have to be a single line input. If you have a line break in them, it will be ignored. This is because how it's currently implemented, and fixing this would require a proper AST setup when parsing a file.
Currently Importer Marker and Exporter Marker have to be a single line input. If you have a line break in them, it will be ignored. This is because how it's currently implemented, and fixing this would require a proper AST setup when parsing a file.

### Add special annotations `ignore` to skip Importer run
### Add special markers `ignore` to skip Importer run

When having an automation such as `find . -name '*.md' -exec importer generate {} \;`, you may want to skip some files.

This shouldn't skip Export Marker handling, though.
This shouldn't skip Exporter Marker handling, though.

### Add `diff` command

Expand Down
46 changes: 23 additions & 23 deletions docs/getting-started/examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,41 +2,41 @@

<!-- == export: simple / begin == -->

Let's see what Importer does with the file in this repository [`./testdata/simple-before.md`](https://raw.githubusercontent.com/upsidr/importer/main/testdata/simple-before.md).
Let's see what Importer does with the file in this repository [`./testdata/markdown/simple-before.md`](https://raw.githubusercontent.com/upsidr/importer/main/testdata/simple-before.md).

```markdown
# Simple Markdown Test

<!-- == imptr: lorem / begin from: ../docs/template/_lorem.md#5~12 == -->
<!-- == imptr: lorem / begin from: ../../docs/template/_lorem.md#5~12 == -->

Any content here will be removed by Importer.

<!-- == imptr: lorem / end == -->

Content after annotation is left untouched.
Content after marker is left untouched.
```

When you run `importer purge ./testdata/simple-before.md`:
When you run `importer purge ./testdata/markdown/simple-before.md`:

```bash
$ importer purge ./testdata/simple-before.md
$ cat ./testdata/simple-before.md
$ importer purge ./testdata/markdown/simple-before.md
$ cat ./testdata/markdown/simple-before.md
# Simple Markdown Test

<!-- == imptr: lorem / begin from: ../docs/template/_lorem.md#5~12 == -->
<!-- == imptr: lorem / begin from: ../../docs/template/_lorem.md#5~12 == -->
<!-- == imptr: lorem / end == -->

Content after annotation is left untouched.
Content after marker is left untouched.
```

When you run `importer generate ./testdata/simple-before.md`:
When you run `importer generate ./testdata/markdown/simple-before.md`:

```bash
$ importer generate ./testdata/simple-before.md
$ cat ./testdata/simple-before.md
$ importer generate ./testdata/markdown/simple-before.md
$ cat ./testdata/markdown/simple-before.md
# Simple Markdown Test

<!-- == imptr: lorem / begin from: ../docs/template/_lorem.md#5~12 == -->
<!-- == imptr: lorem / begin from: ../../docs/template/_lorem.md#5~12 == -->
"Lorem ipsum dolor sit amet,
consectetur adipiscing elit,
sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Expand All @@ -47,7 +47,7 @@ Excepteur sint occaecat cupidatat non proident,
sunt in culpa qui officia deserunt mollit anim id est laborum."
<!-- == imptr: lorem / end == -->

Content after annotation is left untouched.
Content after marker is left untouched.
```

<!-- == export: simple / end == -->
Expand All @@ -68,7 +68,7 @@ Any content here will be removed by Importer.

<!-- == imptr: lorem / end == -->

Content after annotation is left untouched.
Content after marker is left untouched.
EOF

# Create a file with Lorem Ipsum in a separate file
Expand Down Expand Up @@ -110,30 +110,30 @@ $ importer preview /tmp/importer-example.md
Content Before:
0: # Simple Markdown Test
1:
2: <!-- == imptr: lorem / begin from: ../docs/template/_lorem.md#5~12 == -->
2: <!-- == imptr: lorem / begin from: ../../docs/template/_lorem.md#5~12 == -->
3:
4: Any content here will be removed by Importer.
5:
6: <!-- == imptr: lorem / end == -->
7:
8: Content after annotation is left untouched.
8: Content after marker is left untouched.
---------------

---------------
Content After Purged:
0: # Simple Markdown Test
1:
2: <!-- == imptr: lorem / begin from: ../docs/template/_lorem.md#5~12 == -->
2: <!-- == imptr: lorem / begin from: ../../docs/template/_lorem.md#5~12 == -->
3: <!-- == imptr: lorem / end == -->
4:
5: Content after annotation is left untouched.
5: Content after marker is left untouched.
---------------

---------------
Content After Processed:
1: # Simple Markdown Test
2:
3: <!-- == imptr: lorem / begin from: ../docs/template/_lorem.md#5~12 == -->
3: <!-- == imptr: lorem / begin from: ../../docs/template/_lorem.md#5~12 == -->
4: "Lorem ipsum dolor sit amet,
5: consectetur adipiscing elit,
6: sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Expand All @@ -144,15 +144,15 @@ Content After Processed:
11: sunt in culpa qui officia deserunt mollit anim id est laborum."
12: <!-- == imptr: lorem / end == -->
13:
14: Content after annotation is left untouched.
14: Content after marker is left untouched.
---------------

You can replace the file content with either of the commands below:

- 'importer generate testdata/simple-before.md'
Replace the file content with the processed file, importing all annotated references.
- 'importer purge testdata/simple-before.md'
Replace the file content by removing all data between annotation pairs.
Replace the file content by removing all data between marker pairs.

You can find more with 'importer help'
```
Expand Down Expand Up @@ -182,7 +182,7 @@ The below is how the file would look like after `importer purge` and `importer g
<!-- == imptr: lorem / begin from: ./lorem.md#5~12 == -->
<!-- == imptr: lorem / end == -->

Content after annotation is left untouched.
Content after marker is left untouched.
```

</details>
Expand All @@ -208,7 +208,7 @@ Excepteur sint occaecat cupidatat non proident,
sunt in culpa qui officia deserunt mollit anim id est laborum."
<!-- == imptr: lorem / end == -->

Content after annotation is left untouched.
Content after marker is left untouched.
```

</details>
Loading