Decouple time macro in source path from time spec

[shawncao]

Currently we support roll spec with time MACROs (date, hour, min, second) by specifying MACRO pattern in time spec. I think we should decouple this to get a better flexibility.

For example, below spec should be a legit spec:

test-table:
  ...
  source: s3://xxx/{date}/{hour}/
  time:
      type: column
      column: col2
      pattern: UNIXTIME_MS

This spec is basically asking us to scan s3 file path with supported macros in it, but the time is actually from an existing column. My understanding is we don't support MACRO parsing if we don't specify it in time spec.

cc @chenqin

[chenqin]

True, we should consider extra timestamp from record instead of using batch time macro

for swap, we don't have timestamp...

nebula/src/ingest/SpecRepo.cpp

Line 132 in 701265b

genSpecPerFile(table, version, files, specs, 0);

for roll, we use timebucket to infer batch timestamp aka watermark

nebula/src/ingest/SpecRepo.cpp

Line 160 in 701265b

const auto watermark = now - i * curUnitInSeconds;

nebula/src/ingest/SpecRepo.cpp

Lines 92 to 96 in 701265b

	void genSpecPerFile(const TableSpecPtr& table,
	const std::string& version,
	const std::vector<FileInfo>& files,
	std::vector<std::shared_ptr<IngestSpec>>& specs,
	size_t watermark) noexcept {

[chenqin]

If no objection, will prepare a pr shortly.

[shawncao]

Thanks Chen, I don’t know if you have bandwidth, :) so I made some changes this afternoon to test a dataset of ‘changes.feed’. I’ll wrap it and send a diff.

…

On Tue, Jan 26, 2021 at 10:22 PM Chen Qin ***@***.***> wrote: If no objection, will prepare a pr shortly. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#84 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGLGAZKIW4J3WDXTOALQ2QDS36WMFANCNFSM4WRGERNA> .