Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support noOverwriteGlobs for templates based on react #1191

Closed
wants to merge 69 commits into from
Closed
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
fc81484
Merge pull request #1 from asyncapi/master
lmgyuan Apr 16, 2024
94c04b5
fix noOverwriteGlobs issue #1128
lmgyuan Apr 16, 2024
4bccad3
Merge branch 'fix-noOverwriteGlobs-1128' of https://github.com/lmgyua…
lmgyuan Apr 16, 2024
57ec2ee
give noOverwriteGlobs a default value of []
lmgyuan Apr 16, 2024
8333600
Update integration.test.js
lmgyuan Apr 18, 2024
d467476
handle "no such file/directory" error
lmgyuan Apr 18, 2024
4999885
handle output path error
lmgyuan Apr 18, 2024
19e11df
edit package require
lmgyuan Apr 18, 2024
c0b9ec8
edit generator parameter
lmgyuan Apr 18, 2024
d147b9f
resolved asynchronous issue
lmgyuan Apr 18, 2024
7dd724f
solve asynchronous in test
lmgyuan Apr 18, 2024
1ff0f22
make sure read is executed before checking
lmgyuan Apr 18, 2024
98a1e71
delete unused packages
lmgyuan Apr 18, 2024
e9f5bd0
suppress no-duplicate-string
lmgyuan Apr 18, 2024
695e960
Merge branch 'master' into fix-noOverwriteGlobs-1128
derberg Apr 22, 2024
3bb9e6f
add log.debug to replace console.log
lmgyuan Apr 23, 2024
6aa3b59
add debug parameter
lmgyuan May 14, 2024
380e210
resolve "unresolved names"
lmgyuan May 14, 2024
1b638a2
Merge remote-tracking branch 'upstream/master' into fix-noOverwriteGl…
lmgyuan May 15, 2024
30bf2f6
change index.html to use updated test md
lmgyuan May 16, 2024
a7930b8
update template used for tests
lmgyuan May 16, 2024
2e8dda6
use the right way of writing string with variables in javascript
lmgyuan May 16, 2024
1047cda
Update comments in test/integration.test.js
lmgyuan May 16, 2024
a3c0a86
change comments; add log check; store content in variable
lmgyuan May 16, 2024
95d44f8
change log variable
lmgyuan May 16, 2024
2e6cd81
change output variable
lmgyuan May 16, 2024
36815ce
comment out log for test
lmgyuan May 16, 2024
655d39b
uncomment log test
lmgyuan May 16, 2024
e075fff
mock console.log function
lmgyuan May 16, 2024
a94fd00
comment log
lmgyuan May 20, 2024
cd22f3e
update Integration.test
lmgyuan May 20, 2024
2117147
add existSync
lmgyuan May 20, 2024
6ae0b80
update existsSync
lmgyuan May 20, 2024
33eb3e5
update logMessages, generator, and integration test
lmgyuan May 20, 2024
ca40710
update generator and integration.test
lmgyuan May 20, 2024
332a4ed
update generator and integration.test
lmgyuan May 20, 2024
214387a
update logMessages
lmgyuan May 20, 2024
e5db0f6
update logMessages and integration
lmgyuan May 20, 2024
3505053
update logMessages & integration.test
lmgyuan May 20, 2024
c4f29c2
delete variables
lmgyuan May 20, 2024
038c130
update generator.js and integration test
lmgyuan May 20, 2024
b2fa883
Merge pull request #3 from asyncapi/master
lmgyuan May 20, 2024
15629dc
update to full path
lmgyuan May 20, 2024
5e49121
update generator.js, react.js, integration.test.js
lmgyuan May 21, 2024
a23636a
update generator.js and integration.test
lmgyuan May 21, 2024
55f9ae7
update filtersRegistry, generator, integration.test
lmgyuan May 22, 2024
900faa9
check whether a file path should be overwritten
lmgyuan May 24, 2024
3e7afe7
fixed logging issue
lmgyuan May 24, 2024
b3b29a1
update generator.js and react.js
lmgyuan May 24, 2024
6854f6b
Revert "update generator.js and react.js"
lmgyuan May 24, 2024
fbd41c8
Update package.json
lmgyuan May 24, 2024
09e7882
Revert "fixed logging issue"
lmgyuan May 24, 2024
49c8952
Revert "check whether a file path should be overwritten"
lmgyuan May 24, 2024
eebca77
Revert "update filtersRegistry, generator, integration.test"
lmgyuan May 24, 2024
2f3a3ef
Reapply "update filtersRegistry, generator, integration.test"
lmgyuan May 24, 2024
f5a6251
update package.json & integration.test.js
lmgyuan May 24, 2024
0385f26
eliminated unnecessary comments and codes;
lmgyuan May 24, 2024
be28dea
normalize file path in integration test
lmgyuan May 24, 2024
12ada05
normalize testFilePath in integration.test
lmgyuan May 24, 2024
06848d4
Update integration.test.js
lmgyuan May 28, 2024
1ec96a4
Update integration.test.js
lmgyuan May 28, 2024
61d795a
Update integration.test.js
lmgyuan May 28, 2024
f5180b7
add await to saveContentToFile
lmgyuan Jun 10, 2024
4305a9c
used fspromise
lmgyuan Jun 11, 2024
ce41fe0
Merge branch 'master' into fix-noOverwriteGlobs-1128
lmgyuan Jun 11, 2024
c741598
Merge branch 'asyncapi:master' into master
lmgyuan Jun 11, 2024
2ee44db
test again
lmgyuan Jun 11, 2024
2d77e3c
delete unnecessary declarations in integration test
lmgyuan Jun 25, 2024
47ae19b
Merge branch 'master' into fix-noOverwriteGlobs-1128
lmgyuan Jun 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 16 additions & 16 deletions lib/generator.js
Original file line number Diff line number Diff line change
Expand Up @@ -532,7 +532,7 @@ class Generator {
arbOptions.registry = providedRegistry;
registryUrl = providedRegistry;
}

const domainName = registryUrl.replace(/^https?:\/\//, '');
//doing basic if/else so basically only one auth type is used and token as more secure is primary
if (this.registry.token) {
Expand All @@ -541,7 +541,7 @@ class Generator {
} else if (this.registry.auth) {
authorizationName = `//${domainName}:_auth`;
arbOptions[authorizationName] = this.registry.auth;
}
}

//not sharing in logs neither token nor auth for security reasons
log.debug(`Using npm registry ${registryUrl} and authorization type ${authorizationName} to handle template installation.`);
Expand All @@ -556,14 +556,14 @@ class Generator {
let pkgPath;
let installedPkg;
let packageVersion;

try {
installedPkg = getTemplateDetails(this.templateName, PACKAGE_JSON_FILENAME);
pkgPath = installedPkg && installedPkg.pkgPath;
packageVersion = installedPkg && installedPkg.version;
log.debug(logMessage.templateSource(pkgPath));
if (packageVersion) log.debug(logMessage.templateVersion(packageVersion));

return {
name: installedPkg.name,
path: pkgPath
Expand All @@ -573,38 +573,38 @@ class Generator {
// We did our best. Proceed with installation...
}
}

const debugMessage = force ? logMessage.TEMPLATE_INSTALL_FLAG_MSG : logMessage.TEMPLATE_INSTALL_DISK_MSG;
log.debug(logMessage.installationDebugMessage(debugMessage));

if (isFileSystemPath(this.templateName)) log.debug(logMessage.NPM_INSTALL_TRIGGER);

const arbOptions = {
path: ROOT_DIR,
};
if (this.registry) {
this.initialiseArbOptions(arbOptions);
}

const arb = new Arborist(arbOptions);

try {
const installResult = await arb.reify({
add: [this.templateName],
saveType: 'prod',
save: false
});

const addResult = arb[Symbol.for('resolvedAdd')];
if (!addResult) throw new Error('Unable to resolve the name of the added package. It was most probably not added to node_modules successfully');

const packageName = addResult[0].name;
const packageVersion = installResult.children.get(packageName).version;
const packagePath = installResult.children.get(packageName).path;

if (!isFileSystemPath(this.templateName)) log.debug(logMessage.templateSuccessfullyInstalled(packageName, packagePath));
if (packageVersion) log.debug(logMessage.templateVersion(packageVersion));

return {
name: packageName,
path: packagePath,
Expand Down Expand Up @@ -834,14 +834,14 @@ class Generator {
* @param {String} outputPath Path to the resulting rendered file.
* @param {Object} [extraTemplateData] Extra data to pass to the template.
*/
async renderAndWriteToFile(asyncapiDocument, templateFilePath, outputpath, extraTemplateData) {
async renderAndWriteToFile(asyncapiDocument, templateFilePath, outputPath, extraTemplateData) {
const renderContent = await this.renderFile(asyncapiDocument, templateFilePath, extraTemplateData);
if (renderContent === undefined) {
return;
} else if (isReactTemplate(this.templateConfig)) {
await saveRenderedReactContent(renderContent, outputpath);
await saveRenderedReactContent(renderContent, outputPath, this.noOverwriteGlobs);
} else {
await writeFile(outputpath, renderContent);
await writeFile(outputPath, renderContent);
}
}

Expand Down
52 changes: 36 additions & 16 deletions lib/renderer/react.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
const path = require('path');
const AsyncReactSDK = require('@asyncapi/generator-react-sdk');
const minimatch = require('minimatch');
const {
writeFile
} = require('../utils');
Expand All @@ -8,7 +9,7 @@ const reactExport = module.exports;

/**
* Configures React templating system, this handles all the transpilation work.
*
*
* @private
* @param {string} templateLocation located for thetemplate
* @param {string} templateContentDir where the template content are located
Expand All @@ -23,76 +24,95 @@ reactExport.configureReact = async (templateLocation, templateContentDir, transp

/**
* Renders the template with react and returns the content and meta data for the file.
*
*
* @private
* @param {AsyncAPIDocument} asyncapiDocument
* @param {AsyncAPIDocument} asyncapiDocument
* @param {string} filePath path to the template file
* @param {Object} extraTemplateData Extra data to pass to the template.
* @param {string} templateLocation located for thetemplate
* @param {string} templateContentDir where the template content are located
* @param {string} transpiledTemplateLocation folder for the transpiled code
* @param {Object} templateParams provided template parameters
* @param {boolean} debug flag
* @param {string} originalAsyncAPI
* @param {string} originalAsyncAPI
* @return {Promise<TemplateRenderResult>}
*/
reactExport.renderReact = async (asyncapiDocument, filePath, extraTemplateData, templateLocation, templateContentDir, transpiledTemplateLocation, templateParams, debug, originalAsyncAPI) => {
extraTemplateData = extraTemplateData || {};
filePath = filePath.replace(templateContentDir, path.resolve(templateLocation, transpiledTemplateLocation));
return await AsyncReactSDK.renderTemplate(
filePath,
filePath,
{
asyncapi: asyncapiDocument,
params: templateParams,
originalAsyncAPI,
...extraTemplateData
},
},
debug
);
};

/**
* Save the single rendered react content based on the meta data available.
*
*
* @private
* @param {TemplateRenderResult} renderedContent the react content rendered
* @param {String} outputPath Path to the file being rendered.
* @param {String[]} noOverwriteGlobs globs to check for files that should not be overwritten.
*/
const saveContentToFile = async (renderedContent, outputPath) => {
const saveContentToFile = async (renderedContent, outputPath, noOverwriteGlobs = []) => {
let filePath = outputPath;
// Might be the same as in the `fs` package, but is an active choice for our default file permission for any rendered files.
// Might be the same as in the `fs` package, but is an active choice for our default file permission for any rendered files.
let permissions = 0o666;
const content = renderedContent.content;

// Error handling for undefined content or non-string type
if (content === undefined || typeof content !== 'string') {
throw Error(`Template file not rendered correctly, was rendered as ${content}`);
}

// Adjust file permissions based on metadata, if available
if (renderedContent.metadata !== undefined) {
if (renderedContent.metadata.permissions !== undefined) {
permissions = renderedContent.metadata.permissions;
}

// Replace the base filename with a new filename from metadata if specified
if (renderedContent.metadata.fileName !== undefined) {
const newFileName = renderedContent.metadata.fileName.trim();
const basepath = path.basename(filePath);
filePath = filePath.replace(basepath, newFileName);
}
}

await writeFile(filePath, content, {
mode: permissions
});
// get the final file name of the file
const finalFileName = path.basename(filePath);
// check whether the filename should be ignored based on user's inputs
const shouldOverwrite = !noOverwriteGlobs.some(globExp => minimatch(finalFileName, globExp));

// Write the file only if it should not be skipped
if (shouldOverwrite) {
await writeFile(filePath, content, {
mode: permissions
});
} else {
console.log(`Skipping overwrite for: ${filePath}`);
derberg marked this conversation as resolved.
Show resolved Hide resolved
}
};

/**
* Save the rendered react content based on the meta data available.
*
*
* @private
* @param {TemplateRenderResult[] | TemplateRenderResult} renderedContent the react content rendered
* @param {String} outputPath Path to the file being rendered.
* @param {String[]} noOverwriteGlobs globs to check for files that should not be overwritten.
*/
reactExport.saveRenderedReactContent = async (renderedContent, outputPath) => {
reactExport.saveRenderedReactContent = async (renderedContent, outputPath, noOverwriteGlobs = []) => {
// If the rendered content is an array, we need to save each file individually
if (Array.isArray(renderedContent)) {
return Promise.all(renderedContent.map(content => saveContentToFile(content, outputPath)));
return Promise.all(renderedContent.map(content => saveContentToFile(content, outputPath, noOverwriteGlobs)));
}
return saveContentToFile(renderedContent, outputPath);
// Otherwise, we can save the single file
return saveContentToFile(renderedContent, outputPath, noOverwriteGlobs);
};
26 changes: 26 additions & 0 deletions test/integration.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ const dummySpecPath = path.resolve(__dirname, './docs/dummy.yml');
const refSpecPath = path.resolve(__dirname, './docs/apiwithref.json');
const refSpecFolder = path.resolve(__dirname, './docs/');
const crypto = require('crypto');
const {mkdirSync} = require('fs');
const {exists, writeFile} = require('../lib/utils');
const mainTestResultPath = 'test/temp/integrationTestResult';
//we do not want to download chromium for html-template if it is not needed
process.env['PUPPETEER_SKIP_CHROMIUM_DOWNLOAD'] = true;
Expand All @@ -25,6 +27,7 @@ describe('Integration testing generateFromFile() to make sure the result of the
const outputDir = generateFolderName();
const generator = new Generator('@asyncapi/html-template@0.18.0', outputDir, { forceWrite: true, templateParams: { singleFile: true } });
await generator.generateFromFile(dummySpecPath);
// eslint-disable-next-line sonarjs/no-duplicate-string
const file = await readFile(path.join(outputDir, 'index.html'), 'utf8');
expect(file).toMatchSnapshot();
});
Expand All @@ -48,4 +51,27 @@ describe('Integration testing generateFromFile() to make sure the result of the
const file = await readFile(path.join(outputDir, 'index.html'), 'utf8');
expect(file).toMatchSnapshot();
});

it('should ignore specified files with noOverwriteGlobs', async () => {
const outputDir = generateFolderName();
// Manually create a file to test if it's not overwritten
lmgyuan marked this conversation as resolved.
Show resolved Hide resolved
if (!await exists(outputDir)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually I don't get why we need to validate if it exists, if a line earlier it is created

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const generateFolderName = () => {
    // you always want to generate to new directory to make sure test runs in clear environment
    // normalize the path for cross-platform compatibility
    // return path.normalize(path.resolve(mainTestResultPath, crypto.randomBytes(4).toString('hex')));
    return path.resolve(mainTestResultPath, crypto.randomBytes(4).toString('hex'));
  };

If I understand it correctly, generateFolderName() does not actually generate the directory, but just creates and returns a path string. Thus I manually checks and creates it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that in other tests you have written, you did not check and create the directory. Did you not have directory or path not found issue? If I comment out line 78-80, I have

ENOENT: no such file or directory, open '/Users/yuanyuan/workspace/generator/test/temp/integrationTestResult/6224bfdd/test-file.md'

mkdirSync(outputDir, { recursive: true });
}
// eslint-disable-next-line sonarjs/no-duplicate-string
const testFilePath = path.join(outputDir, 'index.html');
await writeFile(testFilePath, '<script>const initialContent = "This should not change";</script>');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't it better to add the string to variable and not repeat it in line 82?


// based on the html-template documentation, the default output is index.html
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not relevant anymore

const generator = new Generator('@asyncapi/html-template@0.28.0', outputDir, {
forceWrite: true,
derberg marked this conversation as resolved.
Show resolved Hide resolved
noOverwriteGlobs: ['**/index.html']
});

await generator.generateFromFile(dummySpecPath);

// Read the file to confirm it was not overwritten
const fileContent = await readFile(testFilePath, 'utf8');
expect(fileContent).toBe('<script>const initialContent = "This should not change";</script>');
});
});
Loading