This repository has been archived by the owner. It is now read-only.

`HIGHLIGHT_DATADIR` Fails on Windows #24

Closed
tajmone opened this Issue Sep 26, 2017 · 16 comments

Comments

Projects
None yet
2 participants
@tajmone
Contributor

tajmone commented Sep 26, 2017

I'm trying to force Highlight to use my custom lang def files instead of those that come with the installation. I'm using Windows 10 x64, and Highlight x64 (full installation).

Highlight is being invoked by a console app I'm working on. The app defines the env var HIGHLIGHT_DATADIR, pointing to a local folder (absolute path) which contains the following subfolders:

  • langDefs
  • plugins
  • themes

... but Highlight always ends up using the lang def from the installation folder.

NOTE: The app is working fine with Highlight. The only problem I'm facing is trying to use a custom data folder. The console app runs in Git Bash only (doesn't support Windows CMD because of some Bash commands it relies on); but this shouldn't affect Highlight's workings. Also, I've checked that the HIGHLIGHT_DATADIR env var is visible to Highlight at invocation time.

The README file states that:

Configuration scripts are searched in the following directories:

1. ~/.highlight/
2. value of the environment variable HIGHLIGHT_DATADIR
3. user defined directory set with --data-dir (deprecated option)
4. /usr/share/highlight/
5. /etc/highlight/ (location of filetypes.conf)
6. current working directory (fallback)

I'm not sure which are the Windows equivalents of some of those paths. But I've checked my user folders, and the only Highlight related folder I could find was C:\Users\MY_USER_NAME\AppData\Roaming\Highlight (which is empty).

From the CMD, I can see that Highlight is listed in the system PATH env var (C:\Program Files\Highlight) — could this be the reason why it's always reading the lang def from the install folder?

I've also tried using the --data-dir= option in my app's Highlight-invocation options ... same result!

Any advise on this?

@tajmone

This comment has been minimized.

Show comment
Hide comment
@tajmone

tajmone Sep 26, 2017

Contributor

PS: I can't remember if I added manually Highlight's path to the system PATH (ie: to have access to the cli version from anywhere) or if it was Highlight's installer that added it. But this doesn't look like the root of the problem because I've tried adding my custom data folder at the beginning of the sys PATH and nothing changed.

I've tried invoking Highlight from a batch file (without my app) and I still can't get it to work. I've tried:

  • The HIGHLIGHT_DATADIR env var approach;
  • Using the --data-dir= option;
  • Adding to beginning of the sys PATH the path of my custom data folder (so it would be seen before C:\Program Files\Highlight )

The only way I can get a custom syntax to work is with the --config-file option!

Contributor

tajmone commented Sep 26, 2017

PS: I can't remember if I added manually Highlight's path to the system PATH (ie: to have access to the cli version from anywhere) or if it was Highlight's installer that added it. But this doesn't look like the root of the problem because I've tried adding my custom data folder at the beginning of the sys PATH and nothing changed.

I've tried invoking Highlight from a batch file (without my app) and I still can't get it to work. I've tried:

  • The HIGHLIGHT_DATADIR env var approach;
  • Using the --data-dir= option;
  • Adding to beginning of the sys PATH the path of my custom data folder (so it would be seen before C:\Program Files\Highlight )

The only way I can get a custom syntax to work is with the --config-file option!

@andre-simon

This comment has been minimized.

Show comment
Hide comment
@andre-simon

andre-simon Sep 26, 2017

Owner

The current version does not check --data-dir or HL_DATADIR when compiled for windows. Omitting a compiler directive will enable this. Do you need a 32bit or 64bit build?

Owner

andre-simon commented Sep 26, 2017

The current version does not check --data-dir or HL_DATADIR when compiled for windows. Omitting a compiler directive will enable this. Do you need a 32bit or 64bit build?

@tajmone

This comment has been minimized.

Show comment
Hide comment
@tajmone

tajmone Sep 27, 2017

Contributor

I'm currently working on a FOSS documentation project for which I'm writing a custom CMS that levers on Pandoc + PP and Highlight. The project (Windows and Linux) is going to be on GitHub, so end users will have to install the binary dependecies separately from the repo source.

Ideally, it will be up to the users to choose which architecture to use (my app will build for both x32 and x64). That said, the x86_32 bit version should be fine since Highlight is invoked as an independent application, so it doesn't matter which bitness it is.

What I need to know though, is if future Windows builds of Highlight (as downloadable from your website) will support the env var HIGHLIGHT_DATADIR .

I was thinking of adding to my CMS a function that takes care of downloading all binary dependencies, in standalone format, and unpack the required files into the project folder. This will allow finer control over which version of each app (Pandoc, PP, and Highlight) the project will be using — thus ensuring that all documents are build exactly the same. Currently, the CMS already has an .ini file that specifies which versions of these apps are required (strict versioning).

The whole issue with the env var HIGHLIGHT_DATADIR is because I'll be using some custom lang defs which are intended for the project's documents (and might not have any use elsewhere).

Are there some tech problems with support for HIGHLIGHT_DATADIR in Windows builds?

Contributor

tajmone commented Sep 27, 2017

I'm currently working on a FOSS documentation project for which I'm writing a custom CMS that levers on Pandoc + PP and Highlight. The project (Windows and Linux) is going to be on GitHub, so end users will have to install the binary dependecies separately from the repo source.

Ideally, it will be up to the users to choose which architecture to use (my app will build for both x32 and x64). That said, the x86_32 bit version should be fine since Highlight is invoked as an independent application, so it doesn't matter which bitness it is.

What I need to know though, is if future Windows builds of Highlight (as downloadable from your website) will support the env var HIGHLIGHT_DATADIR .

I was thinking of adding to my CMS a function that takes care of downloading all binary dependencies, in standalone format, and unpack the required files into the project folder. This will allow finer control over which version of each app (Pandoc, PP, and Highlight) the project will be using — thus ensuring that all documents are build exactly the same. Currently, the CMS already has an .ini file that specifies which versions of these apps are required (strict versioning).

The whole issue with the env var HIGHLIGHT_DATADIR is because I'll be using some custom lang defs which are intended for the project's documents (and might not have any use elsewhere).

Are there some tech problems with support for HIGHLIGHT_DATADIR in Windows builds?

@tajmone

This comment has been minimized.

Show comment
Hide comment
@tajmone

tajmone Sep 27, 2017

Contributor

ADDENDUM — I was wondering: but if I implement that my CMS app downloads in its own folder the standalone version of Highlight, when it invokes it via command line, will Highlight look for the lanfDefs folder in the location of its binary? ie: even if the user has installed Highlight on the sytem (via setup), will this standalone version look for its data independently?

The CMS app's path is added to system PATH env var (before all other paths) at the beginning of each working session, plus I can make my app invoke the project's Highlight binary with its full path (instead of relying just on it being on system path).

If this is enough to circumvent any conflicts with a pre-existing installation of Highlight, then I might do without the HIGHLIGHT_DATADIR en var — but I'd need to be sure that it won't conflict with any same-named lang defs in ~/.highlight/ folder either! (this is becaue I might be using some tweaked versions of existing lang defs)

Contributor

tajmone commented Sep 27, 2017

ADDENDUM — I was wondering: but if I implement that my CMS app downloads in its own folder the standalone version of Highlight, when it invokes it via command line, will Highlight look for the lanfDefs folder in the location of its binary? ie: even if the user has installed Highlight on the sytem (via setup), will this standalone version look for its data independently?

The CMS app's path is added to system PATH env var (before all other paths) at the beginning of each working session, plus I can make my app invoke the project's Highlight binary with its full path (instead of relying just on it being on system path).

If this is enough to circumvent any conflicts with a pre-existing installation of Highlight, then I might do without the HIGHLIGHT_DATADIR en var — but I'd need to be sure that it won't conflict with any same-named lang defs in ~/.highlight/ folder either! (this is becaue I might be using some tweaked versions of existing lang defs)

@andre-simon

This comment has been minimized.

Show comment
Hide comment
@andre-simon

andre-simon Sep 27, 2017

Owner

The windows build detects the binary path and assumes to find its config file in this directory. So it is no problem to have several Highlight installations work independently. Only Linux and Mac build take care of ~/.highlight/. The PATH variable does not interfere with highlight's file search. The search path collection is done in core/datadir.cpp::initSearchDirectories, in case you want to check priorities.

But future versions will support HIGHLIGHT_DATADIR and --data-dir, so you'll be able to use the same config options on all platforms. I tested it yesterday, there is no technical issue.

Owner

andre-simon commented Sep 27, 2017

The windows build detects the binary path and assumes to find its config file in this directory. So it is no problem to have several Highlight installations work independently. Only Linux and Mac build take care of ~/.highlight/. The PATH variable does not interfere with highlight's file search. The search path collection is done in core/datadir.cpp::initSearchDirectories, in case you want to check priorities.

But future versions will support HIGHLIGHT_DATADIR and --data-dir, so you'll be able to use the same config options on all platforms. I tested it yesterday, there is no technical issue.

@tajmone

This comment has been minimized.

Show comment
Hide comment
@tajmone

tajmone Sep 27, 2017

Contributor

Thanks a lot André, this was really helpful.

For the moment I'll take the approach of having my app install a local copy (portable) of Highlight — which is better, because the CMS should rely on all contributors to use the same releases of all dependencies; and I won't be updating them to newer releases unless there are some new interesting features.

In my source code I can easily differentiate behaviour for Linux/Mac (on one side) and Windows behaviour (on the other), if needed — but it seems that setting the env var HIGHLIGHT_DATADIR on all OSs' is a good idea since it doesn't harm Windows and it will be ready for when the feature is implemented, and it will avoid on Linux/Mac files in ~/.highlight/ . taking precedence.

Anyhow, my project probably won't be published before the next release of Highlight — still many things to fix, and beside the CMS code, there is a lot of work in porting all the old documents to the new system, not to mention the time being consumed by creating a Sass project to handle all stylesheet. But it's already taken shape, and it's starting to look great on the parts that I've already ported to new system.

The search path collection is done in core/datadir.cpp::initSearchDirectories, in case you want to check priorities.

I actualy had already peeked into it ... but without a clear understanding of the rest of the project I couldn't really follow it all the way.

Only Linux and Mac build take care of ~/.highlight/.

... strange, because I do find an empty .highlight folder in my user folder in Windows.

Also, using Highlight inside Git Bash offers the benefit that ~/.highlight/ becomes a valid path to the user's home directory (pointing to the same path as in Windows).

Contributor

tajmone commented Sep 27, 2017

Thanks a lot André, this was really helpful.

For the moment I'll take the approach of having my app install a local copy (portable) of Highlight — which is better, because the CMS should rely on all contributors to use the same releases of all dependencies; and I won't be updating them to newer releases unless there are some new interesting features.

In my source code I can easily differentiate behaviour for Linux/Mac (on one side) and Windows behaviour (on the other), if needed — but it seems that setting the env var HIGHLIGHT_DATADIR on all OSs' is a good idea since it doesn't harm Windows and it will be ready for when the feature is implemented, and it will avoid on Linux/Mac files in ~/.highlight/ . taking precedence.

Anyhow, my project probably won't be published before the next release of Highlight — still many things to fix, and beside the CMS code, there is a lot of work in porting all the old documents to the new system, not to mention the time being consumed by creating a Sass project to handle all stylesheet. But it's already taken shape, and it's starting to look great on the parts that I've already ported to new system.

The search path collection is done in core/datadir.cpp::initSearchDirectories, in case you want to check priorities.

I actualy had already peeked into it ... but without a clear understanding of the rest of the project I couldn't really follow it all the way.

Only Linux and Mac build take care of ~/.highlight/.

... strange, because I do find an empty .highlight folder in my user folder in Windows.

Also, using Highlight inside Git Bash offers the benefit that ~/.highlight/ becomes a valid path to the user's home directory (pointing to the same path as in Windows).

@andre-simon

This comment has been minimized.

Show comment
Hide comment
@andre-simon

andre-simon Sep 28, 2017

Owner

Maybe you created .highlight while testing? ;) The CLI build does not create directories, the GUI stores its INI file in HOME (if not started in portable mode).
I will try running it in Git Bash... in general I don't mind to add this path it as well.

Owner

andre-simon commented Sep 28, 2017

Maybe you created .highlight while testing? ;) The CLI build does not create directories, the GUI stores its INI file in HOME (if not started in portable mode).
I will try running it in Git Bash... in general I don't mind to add this path it as well.

@tajmone

This comment has been minimized.

Show comment
Hide comment
@tajmone

tajmone Sep 29, 2017

Contributor

I surely didn't create the .highlight folder on purpose, maybe it was created as a result of some operations I did with Highlight?.

I use both Highlight (GUI and CLI) installed version and standalone (installed is always latest version, portable version an older one).

Contributor

tajmone commented Sep 29, 2017

I surely didn't create the .highlight folder on purpose, maybe it was created as a result of some operations I did with Highlight?.

I use both Highlight (GUI and CLI) installed version and standalone (installed is always latest version, portable version an older one).

@andre-simon

This comment has been minimized.

Show comment
Hide comment
@andre-simon

andre-simon Oct 2, 2017

Owner

Maybe remove it and check if it comes back ;) Except for the makefiles there are no output operations which create directories (except the GUI to store its settings).

Owner

andre-simon commented Oct 2, 2017

Maybe remove it and check if it comes back ;) Except for the makefiles there are no output operations which create directories (except the GUI to store its settings).

@tajmone

This comment has been minimized.

Show comment
Hide comment
@tajmone

tajmone Oct 11, 2017

Contributor

But future versions will support HIGHLIGHT_DATADIR and --data-dir, so you'll be able to use the same config options on all platforms. I tested it yesterday, there is no technical issue.

After some testing and considerations, I've decided that having my app install a portable version of Highlight is going to be too complex to handle for all platforms (especially for Mac, since I don't own one).

Instead, I've implemented the usage of the --config-file=<file> option, making my app point to a subfolder with my custom lang defs. When Highlight for Win will support HIGHLIGHT_DATADIR override, I'll just change to code accordingly.

Chances are that Highlight will be updated before my project goes online (even in beta). And in any case it's going to require just a small code change on my side (unlike writing the code to install/update portable Highlight for three OSs).

I preferr that users handle their Highlight installation separately, and I won't enforce a strict version-check, just a minimal-version check (eg: if next HL release will support the HIGHLIGHT_DATADIR feature, I'll just impose a min-versin check for HL >= 3.40). I'm quite confident that future releases of Highlight won't break backward compatiblity with lang defs, and since I'll be using custom definition files from the project folder I shouldn't worry to much of users using higher versions than the project-recommended one.

Q. About HL Version Scheme

May I ask you how Highlight versioning scheme works? In v3.39, I guess the 3 represents the MAJOR version, but shoudl the 39 part be interpreted? If I where to implement some version constraints scheme, how could I ensure that newer versions are 100% backward compatible with a given version's lang defs?

Contributor

tajmone commented Oct 11, 2017

But future versions will support HIGHLIGHT_DATADIR and --data-dir, so you'll be able to use the same config options on all platforms. I tested it yesterday, there is no technical issue.

After some testing and considerations, I've decided that having my app install a portable version of Highlight is going to be too complex to handle for all platforms (especially for Mac, since I don't own one).

Instead, I've implemented the usage of the --config-file=<file> option, making my app point to a subfolder with my custom lang defs. When Highlight for Win will support HIGHLIGHT_DATADIR override, I'll just change to code accordingly.

Chances are that Highlight will be updated before my project goes online (even in beta). And in any case it's going to require just a small code change on my side (unlike writing the code to install/update portable Highlight for three OSs).

I preferr that users handle their Highlight installation separately, and I won't enforce a strict version-check, just a minimal-version check (eg: if next HL release will support the HIGHLIGHT_DATADIR feature, I'll just impose a min-versin check for HL >= 3.40). I'm quite confident that future releases of Highlight won't break backward compatiblity with lang defs, and since I'll be using custom definition files from the project folder I shouldn't worry to much of users using higher versions than the project-recommended one.

Q. About HL Version Scheme

May I ask you how Highlight versioning scheme works? In v3.39, I guess the 3 represents the MAJOR version, but shoudl the 39 part be interpreted? If I where to implement some version constraints scheme, how could I ensure that newer versions are 100% backward compatible with a given version's lang defs?

@andre-simon

This comment has been minimized.

Show comment
Hide comment
@andre-simon

andre-simon Oct 11, 2017

Owner

I think the next release will be sometime in November. Currently there are only minor updates on hold.

There will be no major changes in the 3.x line. The Lua file spec was always enhanced, older highlight versions ignore new elements. Since even Debian adopted the 3.x release you can assume any highlight installation will be a 3.x.

Owner

andre-simon commented Oct 11, 2017

I think the next release will be sometime in November. Currently there are only minor updates on hold.

There will be no major changes in the 3.x line. The Lua file spec was always enhanced, older highlight versions ignore new elements. Since even Debian adopted the 3.x release you can assume any highlight installation will be a 3.x.

@tajmone

This comment has been minimized.

Show comment
Hide comment
@tajmone

tajmone Oct 28, 2017

Contributor

Thanks for the 3.40 update implementing support for HIGHLIGHT_DATADIR on Windows.

I'm starting to change my app and scripts to use this feature on Win, and managed to use a custom data dir, but not via the HIGHLIGHT_DATADIR env var, but only using the --data-dir option.

I'm not sure if it's me that I'm doing something wrong, or if it's because I'm invoking Highlight from Git Bash (the latter is unlikely, though)

Just HIGHLIGHT_DATADIR Fails

I've set the HIGHLIGHT_DATADIR env var to point to my project's highlight-data folder, with this exact value:

 D:\GitHub\PB-Archives-New\pb-archives-test\source\_butler_\highlight-data\

Inside the highlight-data folder, there is a langDefs subfolder containing my custom langdefs — right now, just the purebasic.lang file, which is a copy of the one from HL install folder except for some keywords that I tweaked for testing if HIGHLIGHT_DATADIR ovverride is working.

But when I build the project, Highlight is still using the langdefs from the installation folder instead of those from my highlight-data folder!

I've checked that the env var HIGHLIGHT_DATADIR is visible from the script invoking Highlight, and it is.

But --data-dir=$HIGHLIGHT_DATADIR Works!

With these same settings, if I add the --data-dir=$HIGHLIGHT_DATADIR option to Highlight invocation, everything works fine: my tweaked language definition overrides the one in the setup folder:

https://github.com/tajmone/pb-archives-test/blob/f2f6072/_butler_/macros/Highlight.pp#L109

<pre class="!ifdef(3)(\1 \3)(\1)"!ifdef[4]~~~ title="\4"~~~><code class="!ifdef(3)(\1 \3)(\1)"!ifdef[4]~~~ title="\4"~~~>\sh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cat <<'EOF' | highlight -f -S \1 --no-trailing-nl --validate-input --data-dir=$HIGHLIGHT_DATADIR \2
\5
EOF
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</code></pre>

Why isn't the presence of HIGHLIGHT_DATADIR env var sufficient?

The --data-dir option now works on Windows (before it didn't), and to me it's 100% fine, except that it's listed as deprecated in HL's help. Does it mean it will dropped at some point?

Possibly...

I think that the failure is due to context.

I open Bash, inoke my CMS App, the App sets the env var HIGHLIGHT_DATADIR, then it invoked PP, which launches a sh script that invokes Highlight. (quite a long chain!)

In the actual sh script (launched by PP), HIGHLIGHT_DATADIR is visible (I can echo $HIGHLIGHT_DATADIR correctly, in the same script, right before invoking Highlight).

But my guess is that when Highilight is launched, it queries the OS for the HIGHLIGHT_DATADIR env var — and it could be that it doesn't see it because it was set in the Bash context, not in CMD, nor system wide.

My CMS App sets the env var in a way that is visible to the apps it invokes directly (PP in this case), but I wouldn't be surprised if inheritance of the context is lost when Highlight starts.

I've also tried setting HIGHLIGHT_DATADIR in the Bash session, before launching my App; but it didn't make any difference (PP's sh script could see it as before, but Highlight apparently ignored it).

I'll have to do some tests from a CMD session, to see if Highlight sees it from there.

Or I'll have to find a way to make HIGHLIGHT_DATADIR en var visible system wide for the duration of the current Bash session.

Contributor

tajmone commented Oct 28, 2017

Thanks for the 3.40 update implementing support for HIGHLIGHT_DATADIR on Windows.

I'm starting to change my app and scripts to use this feature on Win, and managed to use a custom data dir, but not via the HIGHLIGHT_DATADIR env var, but only using the --data-dir option.

I'm not sure if it's me that I'm doing something wrong, or if it's because I'm invoking Highlight from Git Bash (the latter is unlikely, though)

Just HIGHLIGHT_DATADIR Fails

I've set the HIGHLIGHT_DATADIR env var to point to my project's highlight-data folder, with this exact value:

 D:\GitHub\PB-Archives-New\pb-archives-test\source\_butler_\highlight-data\

Inside the highlight-data folder, there is a langDefs subfolder containing my custom langdefs — right now, just the purebasic.lang file, which is a copy of the one from HL install folder except for some keywords that I tweaked for testing if HIGHLIGHT_DATADIR ovverride is working.

But when I build the project, Highlight is still using the langdefs from the installation folder instead of those from my highlight-data folder!

I've checked that the env var HIGHLIGHT_DATADIR is visible from the script invoking Highlight, and it is.

But --data-dir=$HIGHLIGHT_DATADIR Works!

With these same settings, if I add the --data-dir=$HIGHLIGHT_DATADIR option to Highlight invocation, everything works fine: my tweaked language definition overrides the one in the setup folder:

https://github.com/tajmone/pb-archives-test/blob/f2f6072/_butler_/macros/Highlight.pp#L109

<pre class="!ifdef(3)(\1 \3)(\1)"!ifdef[4]~~~ title="\4"~~~><code class="!ifdef(3)(\1 \3)(\1)"!ifdef[4]~~~ title="\4"~~~>\sh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cat <<'EOF' | highlight -f -S \1 --no-trailing-nl --validate-input --data-dir=$HIGHLIGHT_DATADIR \2
\5
EOF
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</code></pre>

Why isn't the presence of HIGHLIGHT_DATADIR env var sufficient?

The --data-dir option now works on Windows (before it didn't), and to me it's 100% fine, except that it's listed as deprecated in HL's help. Does it mean it will dropped at some point?

Possibly...

I think that the failure is due to context.

I open Bash, inoke my CMS App, the App sets the env var HIGHLIGHT_DATADIR, then it invoked PP, which launches a sh script that invokes Highlight. (quite a long chain!)

In the actual sh script (launched by PP), HIGHLIGHT_DATADIR is visible (I can echo $HIGHLIGHT_DATADIR correctly, in the same script, right before invoking Highlight).

But my guess is that when Highilight is launched, it queries the OS for the HIGHLIGHT_DATADIR env var — and it could be that it doesn't see it because it was set in the Bash context, not in CMD, nor system wide.

My CMS App sets the env var in a way that is visible to the apps it invokes directly (PP in this case), but I wouldn't be surprised if inheritance of the context is lost when Highlight starts.

I've also tried setting HIGHLIGHT_DATADIR in the Bash session, before launching my App; but it didn't make any difference (PP's sh script could see it as before, but Highlight apparently ignored it).

I'll have to do some tests from a CMD session, to see if Highlight sees it from there.

Or I'll have to find a way to make HIGHLIGHT_DATADIR en var visible system wide for the duration of the current Bash session.

@tajmone

This comment has been minimized.

Show comment
Hide comment
@tajmone

tajmone Oct 28, 2017

Contributor

Since I'm using the --data-dir option, maybe I should use a differently named env var — since HIGHLIGHT_DATADIR works on Linux and Mac, maybe I should avoid using both (could it cause conflicts when both work?), and just use another name to the env var that I pass to --data-dir option, eg:

--data-dir=$MY_CUSTOM_HIGHLIGHT_DATADIR

... and rely on the --data-dir option only?

What's your advise on this?

Contributor

tajmone commented Oct 28, 2017

Since I'm using the --data-dir option, maybe I should use a differently named env var — since HIGHLIGHT_DATADIR works on Linux and Mac, maybe I should avoid using both (could it cause conflicts when both work?), and just use another name to the env var that I pass to --data-dir option, eg:

--data-dir=$MY_CUSTOM_HIGHLIGHT_DATADIR

... and rely on the --data-dir option only?

What's your advise on this?

@andre-simon

This comment has been minimized.

Show comment
Hide comment
@andre-simon

andre-simon Oct 29, 2017

Owner

The search path list starts with the value of --data-dir or a default value, which happens to be the app path on windows. So the env var only works as intended for new files, existing files in the installation directory cannot be "overridden" (the first found config file wins). The option --print-config shows the search directories.

You do not need to define a new env variable name, as --data-dir has higher priority.

I have no plans to remove --data-dir, I just considered it redundant some time ago so "deprecated" was added to have a chance to clean up later.

Owner

andre-simon commented Oct 29, 2017

The search path list starts with the value of --data-dir or a default value, which happens to be the app path on windows. So the env var only works as intended for new files, existing files in the installation directory cannot be "overridden" (the first found config file wins). The option --print-config shows the search directories.

You do not need to define a new env variable name, as --data-dir has higher priority.

I have no plans to remove --data-dir, I just considered it redundant some time ago so "deprecated" was added to have a chance to clean up later.

@tajmone

This comment has been minimized.

Show comment
Hide comment
@tajmone

tajmone Oct 30, 2017

Contributor

Thanks, this clarified the issue. It makes sense.

The --print-config tip also helped a lot, it allowed me to play with these values and see actual restults.

It's actually good that HIGHLIGHT_DATADIR env var and --data-dir option have different priority and behaviour — it allows different uses. The --data-dir option works great now on Win, and it's sparing me from having to override my tweaked and custom langdefs on a one-by-one basis.

I'll write down a few notes on this topic and publish them in a dedicated page on my Wiki.

Contributor

tajmone commented Oct 30, 2017

Thanks, this clarified the issue. It makes sense.

The --print-config tip also helped a lot, it allowed me to play with these values and see actual restults.

It's actually good that HIGHLIGHT_DATADIR env var and --data-dir option have different priority and behaviour — it allows different uses. The --data-dir option works great now on Win, and it's sparing me from having to override my tweaked and custom langdefs on a one-by-one basis.

I'll write down a few notes on this topic and publish them in a dedicated page on my Wiki.

@tajmone

This comment has been minimized.

Show comment
Hide comment
@tajmone

tajmone Nov 18, 2017

Contributor

Added Wiki Page on the topic:

https://github.com/tajmone/highlight/wiki/Custom-Data-Folders

(better late than never 🙄!!)

Contributor

tajmone commented Nov 18, 2017

Added Wiki Page on the topic:

https://github.com/tajmone/highlight/wiki/Custom-Data-Folders

(better late than never 🙄!!)

netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this issue May 3, 2018

Update to 3.43. From the changelog:
-updated astyle code to release 3.1 (Rev. 655)
-added webkit reformatting style
-improved several language definitions
-fixed Matlab string recognition (andre-simon/highlight#61)
-fixed Autohotkey escape sequence recognition (thanks to Klaus Daube)
-added excel.lang (andre-simon/highlight#60)
-improved Qt pro file (andre-simon/highlight#59)
-CLI: added --reformat-option (andre-simon/highlight#62)
-CLI: added --line-range (andre-simon/highlight#64)
-GUI: added Bulgarian translation  (thanks to Georgi Sotirov)

-fixed HL_OUTPUT in Lua state for HL_FORMAT_XHTML and HL_FORMAT_TRUECOLOR values
-fixed lib-shared make target
-updated astyle code to release 3.1.0 beta
-added Polygen and EBNF2 syntax definitions (thanks to Tristano Ajmone)
-added pywal terminal colouring template in extras/pywal
-added reformatting style ratliff (replaces banner)
-added extras/langDefs-resources/cleanslate.lang (thanks to Tristano Ajmone)
-improved Perl6 compatibility
-improved PHP string interpolation
-improved Haskell definition (andre-simon/highlight#52)
-CLI: added --canvas option to define background color padding in ANSI output
  (andre-simon/highlight#40)
-GUI: added French translation (thanks to Antoine Belvire)
-GUI: added Scripts tab (suggested by Tristano Ajmone)
-GUI: minor bugfixes

-renamed examples directory to extras
-line anchors (-a) are attached as id attribute to the first span or li tag in
  HTML output (andre-simon/highlight#36)
-renamed ID prefix in outhtml_codefold plug-in to be compatible with -a IDs
-added fstab.lang and added anacrontab in filetypes.conf
-removed references to OutputType::HTML32
-added extras/css-themes and extras/langDefs-resources
  (thanks to Tristano Ajmone)
-CLI: removed deprecated indicator of --data-dir option
-CLI: added --no-version-info option
-GUI: fixed initial theme selection
-GUI: added "Omit version info comment" option
-GUI: added "Copy with MIME type" option for HTML output
  (andre-simon/highlight#32)

-fixed Ruby string parsing (thanks to Jens Schleusener)
-fixed segfault on sparc64 (patch by James Clarke)
-fixed PureBasic definition (andre-simon/highlight#25)
-added CMake definition (andre-simon/highlight#20)
-added email definition (andre-simon/highlight#21)
-linked scm suffix to lisp definition
  (andre-simon/highlight#22)
-W32 CLI: support HIGHLIGHT_DATADIR and --data-dir options
  (andre-simon/highlight#24)
-revised documentation

-added syntax for Docker and Elixir
-improved HTML, Julia, Kotlin and Smalltalk syntax definitions
-GUI: added "Paste, Convert and Copy" button
  (https://sourceforge.net/p/syntaxhighlight/support-requests/4/)

-fixed Bash variable highlighting issue
-updated astyle code to release 3.0.1 (https://sourceforge.net/p/astyle/bugs/438)
-added bash_ref_man7_org.lua plugin

-fixed Perl string highlighting issue
-fixed highlighting if a line continues after the nested code delimiter
-updated astyle code to release 3.0
-added examples/pandoc (thanks to Tristano Ajmone)
-added syntax mapping for markdown
  (andre-simon/highlight#11)
-added syntax mapping for clj
  (andre-simon/highlight#15)
-improved Java definition (andre-simon/highlight#13)
-added theme to JSON converter in  examples/json
  (andre-simon/highlight#8)
-CLI: added support for environment variable HIGHLIGHT_OPTIONS
  (andre-simon/highlight#17)

-fixed code folding plugin to support more Ruby conditional modifiers
  (thanks to Jens Schleusener)
-fixed Perl quoted string highlighting (thanks to Jens Schleusener)
-added new GeneratorOverride syntax definition parameter
-added Filenames parameter in filetypes.conf to assign input filenames
  to syntax types (suggested by Andy)
-added FASM definition and edit-fasm theme (thanks to Tristano Ajmone)
-added outhtml_ie7_webctrl plug-in (suggested by Tristano Ajmone)
-GUI: file extensions can be configured for multiple languages,
  triggers syntax selection prompt
-GUI: added Italian translation (thanks to Tristano Ajmone)

-fixed code folding plugin to support Ruby conditional modifiers
-fixed JSON definition (thanks to Timothee Cour)
-fixed output of unknown syntax warning with applied force switch
  (thanks to Andy)
-added state trace parameter to Decorate plug-in function
-added GDScript definition and edit-godot theme (thanks to Tristano Ajmone)
-updated SWIG code samples
-updated Artistic Style lib (SVN Rev. 553)
-revised docs
-CLI: fixed creation of hidden files if output filename is prepended by its
  input path
-CLI: added switch --stdout (https://sourceforge.net/p/syntaxhighlight/bugs/14)

-fixed segfault with --skip applied on a single file input list
  (thanks to Jens Schleusener)
-added support for Python 3.6 syntax
-added Github and Sourceforge themes

-fixed highlighting of nested section delimiters
-fixed PHP definition (thanks to Christoph Burschka)
-fixed font family declaration in SVG
-fixed user defined encoding in ODT
-fixed unnecessary output of style file with --inline-css
  (thanks to Jens Schleusener)
-added vimscript language definition (thanks to Max Christian Pohle)
-added Coffeescript language definition (thanks to Jess Austin)
-added PureBasic definition and theme (thanks to Tristano Ajmone)
-added JSX language definition (suggested by Max Stoiber)
-added PO translation definition
-added plug-in outhtml_add_figure.lua
-updated js definition
-updated Artistic Style lib (SVN Rev. 521)
-improved various color themes and syntax definitions

-added support for true color escape codes (--out-format truecolor)
-fixed xterm256 output for paging with less (thanks to Fylwind)
-fixed operator regex in rnc.lang, crk.lang and yaml.lang (thanks to Joe Klauza)
-added Pony and Whiley definitions
-updated Ceylon, Julia and TypeScript definitions
-added Go, AutoHotKey, TypeScript and R to the foldable list in the
  outhtml_codefold.lua plug-in
-removed plugins/bash_ref_linuxmanpages_com.lua
-GUI: fixed README, ChangeLog and License file paths on Linux

-revised documentation
-GUI: fixed minor layout issues

-the data directory can be defined with the HIGHLIGHT_DATADIR environment variable
-fixed RTF output of UTF-8 input; needs input encoding set to utf-8
  (thanks to Kamigishi Rei)
-fixed XML comment recognition (thanks to Mani)
-data search directories were appended to the result of --list-scripts
-revised older syntax definitions
-updated base URLs of bash_ref_linuxmanpages and cpp_ref_qtproject plug-ins
-GUI: added system copy and paste shortcuts for clipboard functions
  (suggested by Kamigishi Rei)

-added Ansible Yaml definition (thanks to Raphael Droz)
-added Chapel definition (thanks to Lydia Duncan)
-fixed gcc 6 warnings about deprecated auto_ptr usage
-src/makefile: added -std=c++11 because of auto_ptr to unique_ptr transition
  (thanks to Jens Schleusener)
-GUI: fixed style file output if "write to source directory" option is
  checked (thanks to Jim Pattee)

-added support of Pascal, Lua, Ruby and C# regions in outhtml_codefold.lua
-improved outhtml_codefold.lua to handle embedded languages
-added string delimiters in the Ruby definition
-added new AssertEqualLength flag in string section of language definitions
-improved heredoc parsing
-fixed Lua multiline string recognition
-improved SVG whitespace output (patch by Paul de Vrieze)
-added Nim and mIRC Scripting definitions

-improved outhtml_codefold.lua to ignore brackets on the same line
-added RTF output to mark_lines.lua
-fixed Powershell and NSIS definitions
-added JSON and Github Markdown definitions
-CLI: added --keep-injections option to force plugin injection output with -f
-GUI: added keep injections checkbox
-GUI: fixed crash after removing selected plugins

-added HL_REJECT state to be used in a OnStateChange function
-added DecorateLineBegin and DecorateLineEnd hooks
-added mark_lines.lua, outhtml_codefold.lua, comment_links.lua plug-ins
-fixed font face in ODT output
-fixed Operators parameter in frink.lang and oorexx.lang
-fixed regular expression parsing within strings for JS, Perl and Ruby
-CLI: added --page-color option to include a page color in RTF output
-GUI: added RTF page color checkbox

-added new SVG definition to support embedded scripting
-improved js.lang, css.lang, scss.lang, less.lang, tsql.lang
-modified HTML ordered list output to work better with new plug-ins
-renamed plug-in variable HL_INPUT_FILE to HL_PLUGIN_PARAM
-CLI: renamed --plug-in-read option to --plug-in-param
-GUI: updated plug-in parameter label and tool-tips
-GUI: fixed minor issues

-fixed TeX output for cweb documents (patch by Ingo Krabbe)
-fixed string interpolation in bat.lang
-added reduce_filesize.lua, outhtml_add_shadow.lua,
  outhtml_add_background_svg.lua, outhtml_add_background_stripes.lua,
  outhtml_add_line.lua plug-ins
-added TCL extension in examples/tcl
-added kotlin.lang, nginx.lang and julia.lang
-updated php.lang to include version 7 keywords
-updated ceylon.lang to include version 1.2 keywords
-updated scripts in examples directory
-CLI: style-infile option marked as deprecated
-GUI: shortened paths in file input lists

-added rs.lang
-added conf.lang (thanks to Victor Ananjevsky)
-added some extensions in filetypes.conf (patch by Victor Ananjevsky)
-fixed Matlab definition and style (thanks to Justin Pearson)
-CLI: fixed --list-scripts with unknown argument (thanks to Jens Schleusener)

-updated astyle code to release 2.05.1
-fixed shebang recognition (thanks to Victor Ananjevsky)
-GUI: added option to define line numbering start

-added support for LESS, SASS and Stylus CSS processors (suggested by Marcel Bischoff)
-added support for Lua 5.3, removed LUA52 makefile option
-fixed heredoc matching in perl.lang (thanks to cornucopia)
-fixed Haskell lang (thanks to Daan Michiels)
-fixed RNC lang (thanks to Daan Michiels)
-fixed regex pattern in js.lang

-updated astyle code to release 2.05
-added astyle reformatting style vtk

-added bold, italic and underline attributes to xterm256 ANSI output
  (patch by Andrew Fuller)
-fixed assembler mapping in filetypes.conf (thanks to Jens Schleusener)
-added Swift definition
-improved ASP, F#, OCaml and Lisp syntax definitions
-added interpolation patterns to several definitions
-updated base URLs in cpp_ref_gtk_gnome and cpp_ref_qtproject plug-ins
-CLI: added Pango markup output option (patch by Dominik Schmidt)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.