smoke tests fail - no idea how to investiate...

[tripodsan]

the smoke tests are currently failing, I checked out project-helix.io, but on the local checkout, the tests pass.

@kptdobe can you teach us, how to investigate the failing smoke tests?

[rofe]

I created adobe/helix-continuous#17 before seeing this one - feel free to duplicate as you see fit

[tripodsan]

ok, I tried the following:

1. checkout helix-continuous
2. clone helix-cli:

git clone --branch master --single-branch --depth 1 https://github.com/adobe/helix-cli.git

3. setup gdm

cd scripts/gdm; npm install; cd ..

4. run gdm (would be nice it if would show a summary of the used modues)

cd helix-cli; node ../scripts/gdm/index.js; cd ..

5. clone project-helix.io and install:

git clone --single-branch --depth 1 -b master https://github.com/adobe/project-helix.io.git
cd project-helix.io; npm install; cd ..

6. run the smoke tests:

cd project-helix.io; env HLX_SMOKE_EXEC='node ../helix-cli/index.js --log-level warn' npx mocha --exit test/smoke/*
Running smoke test using: node ../helix-cli/index.js --log-level warn

fails!

[tripodsan]

it seems to be an issue with the attribute value output in the htl engine:

            <!-- include ${content.nav}.summary.html-->
            <xesi:include src="${content.nav}.summary.html" />

produces:

            <!-- include &#x2f;SUMMARY.summary.html-->
            <xesi:include src=".summary.html" />

[tripodsan]

adobe/htlengine#64

[tripodsan]

@kptdobe once problem of the smoke tests is, that GDM uses the master of all @adobe dependencies. even if not referenced in the dependencies of the tested project.

eg:

htlengine is a dependency of parcel-plugin-htl but not of helix-cli, but GMD pulls the latest, even if parcel-plugin-htl is not smoke-tested:

during smoke test:

$ npm ls @adobe/htlengine
@adobe/helix-cli@3.2.2 helix-continuous/helix-cli
├── @adobe/htlengine@2.3.0  (github:adobe/htlengine#b8875c637d629620d4bac60751fa2d7aaf2d3141)
└─┬ @adobe/parcel-plugin-htl@2.1.9 (github:adobe/parcel-plugin-htl#73f87d97083361a46a082cc8dc259eb58a5664d9)
  └── UNMET DEPENDENCY @adobe/htlengine@2.2.2

in normal helix-cli:

$ npm ls @adobe/htlengine
@adobe/helix-cli@3.2.2 helix-cli
└─┬ @adobe/parcel-plugin-htl@2.1.9
  └── @adobe/htlengine@2.2.2

[tripodsan]

imminent problem is solved.
keeping open for documentation.

[kptdobe]

hummm.. I would expect that the htlengine dependency gets update to master code also in the parcel-plugin-htl dep: the DGM "should" link sub dependencies of dependencies - see https://github.com/adobe/helix-continuous/blob/master/scripts/gdm/index.js#L149-L170.

The cli that gets created by the GDM is "master" related not "npm version" related: it takes the master branch from all projects, completely independently from npm versions. Goal was to make sure that whatever goes into master from one of our dependencies do not break the cli.
If you release 20 versions of htlengine but only make a PR to upgrade from v2 to v20 in the cli, you do not know which intermediate version broke the cli.
With the continuous release we have now, we could simplify this and only smoke test against last cli master and branch from module being changed (+ sub dependencies update).

I am documenting now the "investigation" process on helix-continuous README.

[kptdobe]

I documented the first level investigation process here: adobe/helix-continuous#18
Direct access to doc section: https://github.com/adobe/helix-continuous/blob/063de67f54fc0f750766cc283c67b862b045cbe0/.circleci/orbs/helix-smoke-tests/README.md#debug