Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Add support for filtering #1

Open
sherriff opened this Issue · 21 comments

3 participants

Erik Drolshammer Trygve Laugstøl cholden14
Erik Drolshammer

Use case:

We have a bunch of Java applications installed using Solaris PKGs. To avoid duplication it would be nice to be able to use templates and just specify the differences using properties.

Example usage:

  1. Have a start-up script template which can be reused by multiple projects. Java options, installation basedir and name of start-up jar file are examples of typical variables.

  2. Reuse manifest files for Solaris SMF. Typical variables that can be substituted are: Path to start-up script and service name.

  3. Reuse action scripts (post-install, pre-remove). Typical variables needed: installation basedir, application name

It seems likely that it is necessary to add configuration option to specify where the different files should be fetched from. E.g. target/unix/files/ and target/unix/scripts instead of src/main/unix/files and src/main/unix/scripts.

See also http://jira.codehaus.org/browse/MUNIX-8 and http://jira.codehaus.org/browse/MUNIX-9.

Trygve Laugstøl
Owner

Would something like this work?

<copyFile>
  <path>goals.txt</path>
  <toFile>/usr/share/hudson/README.txt</toFile>
  <!-- For a single replacement -->
  <replace>
    <pattern>SCRIPT_NAME</pattern>
    <value>app1</value>
  </replace>

  <!-- For multiple replacements -->
  <replacements>
    <replace>
      <pattern>SCRIPT_NAME</pattern>
      <value>app1</value>
    </replace>
    <replace>
      <pattern>SCRIPT_NAME</pattern>
      <value>app1</value>
    </replace>
  </replacements>
</copyFile>

Re "where to copy files from", I think this is covered already by just specifying whatever you want as <path> inside a <copyFile> or <from> inside a <copyDirectory>.

Erik Drolshammer

Comments and questions:

  1. I like the convention over configuration principle that have been used for example to add specific functionality when files are placed in src/main/unix/files and src/main/unix/scripts. A simple boolean to change whether variables in files and scripts folders are replaced or not would then be sufficient. E.g.:

    <configuration>
      <defaults> 
        <filtering>true</filtering>
      </defaults>
    </configuration>
    
  2. Adding support for replacements in copyFile/copyDir seems like a good idea. I guess this can be the underlying implementation for the default filtering option in 1 as well?

  3. I don't quite understand the syntax of the value tag.

    a. I would prefer to be able to turn replacements/filtering on/off with a flag and then expect the replacements that Maven normally would apply. I think this will result in less configuration than specifying all replacements explicitly. This solution would, however, probably solve the problem.

    b. So if I want to use standard maven properties as placeholders to replace it would look something like this?

    in unix-m-p config:

      <replacements>
          <replace>
            <pattern>${application.install.baseDirectory}</pattern>
            <value>/opt/application1</value>
          </replace>
    

    And I'd have to duplicate this information in the pom if the same value should be used other places? E.g.

      <properties>
        <application.install.baseDirectory>/opt/application1</application.install.baseDirectory>
       </properties>
    
  4. I suggest to support replacements only, not a single replacement tag.

Erik Drolshammer

I guess it is a separate issue, but with regards to "where to copy files from". Yes, this is supported by copyFile and copyDirectory. However, I want to use the default "copy rules" that are in effect for src/main/unix/files and scripts as much as possible. The design suggestion in bullet 1 in my previous comment this would be hidden for the user, but the implementation would use a folder containing filtered files instead of the unfiltered templates.

Trygve Laugstøl
Owner

(1) is a good idea that will control filtering on/off on the implicit copyDirectory. Don't know if I need one for file and one for scripts, but I guess the flexibility is good to have.

(2) yes, this will be how everything is implemented.

(3) The value is the exact value that will be used as a value for every regexp match. As outlined in the original example you would do something like this:

<replacements>
  <replace>
    <!--
        ${application.install.baseDirectory} would be replaced by the Maven XML
        reader. We don't want that.
    -->
    <pattern>@application.install.baseDirectory@</pattern>
    <value>${application.install.baseDirectory}</value>
  </replace>
</replacements>

...

<properties>
  <application.install.baseDirectory>/opt/application1</application.install.baseDirectory>
</properties>

But as you say, using the build-in filtering mechanism would be useful. This means that having this:

<properties>
  <application.install.baseDirectory>/opt/application1</application.install.baseDirectory>
</properties>

...

<copyDirectory>
  <path>target/foo</path>
  <to>/usr/share/foo</to>
  <filtering>true</filtering>
</copyDirectory>

would imply this:

<copyDirectory>
  <path>target/foo</path>
  <to>/usr/share/foo</to>
  <replacements>
    <replace>
      <pattern>\${application\.install\.baseDirectory\}</pattern>
      <value>/opt/application1</value>
    </replace>
  </replacements>
</copyDirectory>
Trygve Laugstøl
Owner

An alternative approach which is less work and possibly cleaner as it separates copying from filtering:

<copyDirectory>
  ...
</copyDirectory>
<copyFile>
  ...
</copyFile>

<filter>
  <!-- path refers to installed files -->
  <path>/opt/foo/etc</path>

  <!-- similar to resources plugin, as suggested above -->
  <filtering>true</filtering>

  <!-- as above -->
  <replacements>
    ...
  </replacements>
<filter>

Pros:

  • Less code, I don't have to add support for filtering in 5 tasks (copy directory/file/artifact, extract file/artifact)
  • Cleaner as each task type (copy vs filter) is more specific and hopefully easier to understand

Cons:

  • The existing resources and assembly plugins does copy and filtering in one go which is what people might expect.
  • The Zip package support will
cholden14

Any ETA on Filtering? I am about to have to rewrite our packaging but need filtering to make this particular plugin work.

Trygve Laugstøl
Owner

The code in the repository should work for the zip packaging. Which packaging are you using?

cholden14

Thanks for the reply I am using solaris and rpm's. I have a set of files for the pre and post functions that I want to use as a dependency for each project and have them filtered with application specific information (user, groups, perms).

Any plans to expand the filtering to work with all packages types?

Trygve Laugstøl
Owner

Planned, yes, timeframe, no. Given that people want it makes it more likely, but I don't have any workhours to spend on it now so don't hold your breath. It is doable to implement yourself if you have the time, use the zip code as a start.

Trygve Laugstøl
Owner

@cholden14 Is this still something you need? I'm working on this now.

cholden14
Trygve Laugstøl trygvis referenced this issue from a commit
Trygve Laugstøl o Implementing filtering for rpm and pkg. The pkg part is not very we…
…ll tested yet.

Should fix issue #1.
c5dbf6a
Trygve Laugstøl
Owner

Let me know when you've been able to try it out. I'm working on updating the handbook too.

cholden14
cholden14

Trygvis,

Forgive me but which filtering option did you implement? I mean in terms of basic syntax?

Thanks
Chris

Trygve Laugstøl
Owner

Example syntax:

    <filterFiles>
      <includes>
        <include>/opt/hudson/etc/*</include>
      </includes>
      <excludes>
        <exclude>/opt/hudson/etc/unfiltered.properties</exclude>
      </excludes>
      <!-- Regexes does not work just yet. Need a day more -->
      <regexes>
        <regex>
          <!-- A simple regex -->
          <pattern>MY_PLACEHOLDER</pattern>
          <replacement>MY_VALUE</replacement>
        </regex>
        <regex>
          <!-- A regex using replacements -->
          <pattern>^([-a-z]*)=(/.*)</pattern>
          <replacement>$1=/foo$2</replacement>
        </regex>
      </regexes>
    </filterFiles>

However, while updating the handbook I ran into an issue. Only the keys project.version, project.artifactId and project.groupId will be filtered.

I've more or less fixed the code so that all properties (system, user and project) are used and getting the regexp to work.

cholden14

Trygvis,

Having trouble getting this to build.. keeps wanting to upload to the sonatype. Any suggestions?

cholden14

Nevermind.. idiot typo.. testing the filtering today Trygvis

cholden14

Trygvis,

Ok done some testing and keep getting null pointer exceptions for any of my substitution attempts:

                   <filterFiles>
                        <includes>
                            <include>install/*</include>
                        </includes>
                        <regexes>
                        <regexe>
                            <pattern>${pkg.user}</pattern>
                            <replacement>${user}</replacement>
                        </regexe>
                        </regexes>
                    </filterFiles>

any thoughts?

Trygve Laugstøl
Owner

That filter won't work, ${pkg.user} is not a very good regexp. It should work if you just drop the entire regexes section and make sure ${pkg.user} is defined inside <properties>. If you do not have a regex list will create a set of expressions from the current list of properties.

I tried to document it so if you did read it please let me know what I can improve. It is quite crude yet. Also check out the integration tests for some examples: https://github.com/trygvis/unix-maven-plugin/blob/master/unix-maven-plugin/src/it/test-zip-1/pom4test.xml

Please paste the NPE, that's something that obviously needs fixing :)

cholden14

Trygvis,

OK So I have the following properties specified in what I refer to as my packaging module:


com.theice.tribe:tribe-assembly:zip
some_user
some_group
2044
1501
some_groups
some_groups
start_script
deploy_location
crontab_file
postinstallhook.sh

My files to filter look like so for the properties to filter:

USER=${pkg.user}
GROUP=${pkg.group}
UID=${pkg.uid}
GID=${pkg.gid}

I have also tried:

USER=pkg.user
GROUP=pkg.group
UID=pkg.uid
GID=pkg.gid

In my files to filter what should they places I want substituted to look like? Sorry to be so dense on this but I bet it's something simple I am missing.

I have looked over the examples and I think this should work or maybe I simply can't filter the solaris specific postinstall/preinstall files?

Thanks for your help!

Trygve Laugstøl
Owner

Your file should look like what you have tried:

USER=${pkg.user}

I don't have any tests for pkg, only rpm so if you could try that fast that would be nice. You can also try to run the plugin in debugging mode with -Dmaven.unix.debug and/or -X. Feel free to email me the logs privately.

You should see output like this:

[INFO] Filter Files:
[INFO]  Includes:
[INFO]   /opt/hudson/etc/*
[INFO]  Excludes:
[INFO]   /opt/hudson/etc/filter-2.conf
[INFO]   /opt/hudson/etc/unfiltered.properties
[INFO]  Replacers:
[INFO]   pattern=\Q${version.maven-resources-plugin}\E, replacement=2.5
[INFO]   pattern=\Q${version.maven-project-info-reports-plugin}\E, replacement=2.6
[INFO]   pattern=\Q${version.maven-pmd-plugin}\E, replacement=2.5
[INFO]   pattern=\Q${version.maven-jxr-plugin}\E, replacement=2.3
[INFO]   pattern=\Q${version.maven-javadoc-plugin}\E, replacement=2.9
[INFO]   pattern=\Q${version.maven-deploy-plugin}\E, replacement=2.5
[INFO]   pattern=\Q${version.maven-dependency-plugin}\E, replacement=2.5
....

Note that you should NOT have any <regexp> entries at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.