feat: push expired tiles into job queue and major refactoring by melancholiai · Pull Request #3 · MapColonies/osm2pgsql-wrapper

melancholiai · 2022-03-10T16:11:22Z

Question	Answer
Bug fix	✖
New feature	✔
Breaking change	✔
Deprecations	✔
Documentation	✔
Tests added	✖
Chore	✖

Closes #2

CptSchnitz · 2022-03-14T06:30:23Z

src/commandRunner/OsmCommandRunner.ts

@@ -0,0 +1,74 @@
+import { Logger } from '@map-colonies/js-logger';


file name should be lowercase

CptSchnitz · 2022-03-14T06:38:53Z

README.md

+
+The output of the append command of `osm2pgsql` is an expire list of tiles in requested zoom levels.
+
+The expired tiles list can be uploaded to a source of your choice, `s3` or `queue`.


you dont upload to a queue, you push

it should be mentioned that the queue is pgboss

we should mention that its also possible to use to both.

CptSchnitz · 2022-03-14T06:43:05Z

src/index.ts

+interface IError {
+  // eslint-disable-next-line @typescript-eslint/no-explicit-any
+  error: (...data: any[]) => void;
+}


better name, and move to common

CptSchnitz · 2022-03-14T07:46:19Z

src/containerConfig.ts

+    const httpClientConfig = config.get<object>('httpClient');
+    const httpClient = httpClientFactory(httpClientConfig);


just use axios, there's no need for the factory as its not generic or anything.

CptSchnitz · 2022-03-14T07:47:29Z

src/containerConfig.ts

+      { token: ShutdownHandler, provider: { useValue: shutdownHandler } },
+      { token: CLI_BUILDER, provider: { useFactory: cliBuilderFactory } },
+      { token: CREATE_COMMAND_FACTORY, provider: { useFactory: createCommandFactory } },
+      { token: CREATE_MANAGER_FACTORY, provider: { useFactory: createManagerFactory } },
+      { token: APPEND_COMMAND_FACTORY, provider: { useFactory: appendCommandFactory } },
+      { token: APPEND_MANAGER_FACTORY, provider: { useFactory: appendManagerFactory } },
+      { token: SERVICES.CONFIG, provider: { useValue: config } },
+      { token: SERVICES.LOGGER, provider: { useValue: logger } },
+      { token: SERVICES.TRACER, provider: { useValue: tracer } },
+      { token: SERVICES.HTTP_CLIENT, provider: { useValue: httpClient } },
+      { token: SERVICES.CONFIG_STORE, provider: { useValue: configStore } },
+      { token: EXIT_CODE, provider: { useValue: ExitCodes.SUCCESS } },
+    ];


i think this can be ordered better. config/logger should be first.

CptSchnitz · 2022-03-14T07:50:21Z

helm/README.md


 **postgres:**

+the postgres for osm2pgsql


i think better phrasing is needed.
something like this?
the postgres database target for osm2pgsql?

CptSchnitz · 2022-05-03T05:39:42Z

.gitignore

 jest_html_reporters.html
 reports
+
+local.json


mb local*.json ?
and new line is missing.

CptSchnitz · 2022-05-03T05:45:12Z

helm/README.md

+*append*
 - `cli.append.replicationUrl` - the source of replication
- `cli.append.limit.value` - should limit the amount of appends in a single run
+- `cli.append.limit.enabled` - should limit the amount of appends in a single run


should? use a more decisive wording.

CptSchnitz · 2022-05-03T05:46:23Z

package.json

    "@opentelemetry/api": "1.0.1",
    "@opentelemetry/api-metrics": "0.23.0",
    "@opentelemetry/instrumentation-http": "0.23.0",
+    "@types/mapbox__sphericalmercator": "^1.1.5",


move to dev dep

CptSchnitz · 2022-05-03T05:47:25Z

src/cli/checks/index.ts

+import fsPromises from 'fs/promises';
+import { existsSync } from 'fs';
+import { Arguments } from 'yargs';
+import { isWebUri } from 'valid-url';
+import { AppendArguments, QueueSettings } from '../commands/append/interfaces';
+import { validateBySchema, ValidationResponse } from '../../validation/validator';
+import { LIMIT_SCHEMA, Limit, AppendEntity, QUEUE_SETTINGS_SCHEMA, APPEND_CONFIG_SCHEMA } from '../../validation/schemas';
+import { NOT_FOUND_INDEX } from '../../common/constants';
+import { CreateArguments } from '../commands/create/createFactory';
+import { DumpSourceType } from '../commands/create/constants';
+
+type InvalidHandler<T> = (validationResponse: ValidationResponse<T>) => void;
+type CheckFunc<T> = (args: Arguments<T>) => Promise<boolean> | boolean;
+
+export const limitCheck = (invalidHandler: InvalidHandler<unknown>): CheckFunc<AppendArguments> => {
+  const check: CheckFunc<AppendArguments> = (args) => {
+    const { limit } = args;
+    const validationResponse = validateBySchema<Limit>({ limit }, LIMIT_SCHEMA);
+    invalidHandler(validationResponse);
+    return true;
+  };
+  return check;
+};
+
+export const configCheck = (invalidHandler: InvalidHandler<unknown>): CheckFunc<AppendArguments> => {
+  const check: CheckFunc<AppendArguments> = async (args) => {
+    const { config } = args;
+    const configContent = await fsPromises.readFile(config, 'utf-8');
+    const configContentAsJson: unknown = JSON.parse(configContent);
+    const validationResponse = validateBySchema<AppendEntity[]>(configContentAsJson, APPEND_CONFIG_SCHEMA);
+    invalidHandler(validationResponse);
+    return true;
+  };
+  return check;
+};
+
+export const uploadTargetsCheck = (invalidHandler: InvalidHandler<unknown>): CheckFunc<AppendArguments> => {
+  const check: CheckFunc<AppendArguments> = (args) => {
+    const { uploadTargets } = args;
+    if (uploadTargets.indexOf('queue') !== NOT_FOUND_INDEX) {
+      const { name, minZoom, maxZoom } = args;
+      const request: QueueSettings = {
+        name: name as string,
+        minZoom: minZoom as number,
+        maxZoom: maxZoom as number,
+      };
+      const validationResponse = validateBySchema<QueueSettings>(request, QUEUE_SETTINGS_SCHEMA);
+      invalidHandler(validationResponse);
+    }
+    return true;
+  };
+  return check;
+};
+
+export const dumpSourceCheck = (): CheckFunc<CreateArguments> => {
+  const check: CheckFunc<CreateArguments> = (args) => {
+    const { dumpSourceType, dumpSource } = args;
+
+    const errorPrefix = `provided dump source of type ${dumpSourceType} is not valid`;
+    if (dumpSourceType === DumpSourceType.LOCAL_FILE) {
+      if (!existsSync(dumpSource)) {
+        throw new Error(`${errorPrefix}, ${dumpSource} does not exist locally`);
+      }
+    } else if (isWebUri(dumpSource) === undefined) {
+      throw new Error(`${errorPrefix}, ${dumpSource} is not a valid web uri`);
+    }
+
+    return true;
+  };
+  return check;
+};


some documentation on the functions would be nice.

CptSchnitz · 2022-05-03T05:55:23Z

src/validation/validator.ts

-  public constructor(@inject(SERVICES.LOGGER) private readonly logger: Logger) {
-    this.ajv = new Ajv({ $data: true });
-    ajvKeywords.default(this.ajv);
+export function validateBySchema<T>(content: unknown, schema: JSONSchemaType<T>): ValidationResponse<T> {


its more of an ajv wrapper, than validate by schema.

melancholiai added 2 commits March 3, 2022 16:35

WIP

f50d43a

WIP

2fab1e0

melancholiai requested a review from CptSchnitz March 10, 2022 16:11

WIP

4b2e206

CptSchnitz requested changes Mar 14, 2022

View reviewed changes

melancholiai added 2 commits March 14, 2022 12:01

refactor: changes according to pr

a5a3430

refactor: additional refactoring and bug fixes

89e0e91

melancholiai marked this pull request as ready for review March 14, 2022 15:23

melancholiai added 3 commits March 21, 2022 16:30

refactor: run prettier and added license

6ffaa78

fix: helm typo

dce7b30

fix: current state calculation

2c8e925

CptSchnitz requested changes May 3, 2022

View reviewed changes

refactor: changes according to pr

de65d0c

CptSchnitz approved these changes May 3, 2022

View reviewed changes

melancholiai merged commit 9707577 into master May 3, 2022

		@@ -0,0 +1,74 @@
		import { Logger } from '@map-colonies/js-logger';


		The output of the append command of `osm2pgsql` is an expire list of tiles in requested zoom levels.

		The expired tiles list can be uploaded to a source of your choice, `s3` or `queue`.

		const httpClientConfig = config.get<object>('httpClient');
		const httpClient = httpClientFactory(httpClientConfig);

Conversation

melancholiai commented Mar 10, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants