Some commands not finding the conf correctly #324

chamini2 · 2017-02-21T21:58:17Z

Look at the output of running these two commands one after the other:

$ sqitch status co
# On database co
# Project:  project_name
# Change:   change_sha
# Name:     migration_name
# Tag:      @0.20.0
# Deployed: 2017-01-27 00:33:20 +0000
# By:       ...
#
Undeployed changes:
  * shared/tables/home_configurations/trigger_protect_created_at_column
$ sqitch deploy co
Plan file sqitch.plan does not exist

The sqitch.conf is in the directory these commands were run. status works well, but deploy doesn't. The ones I have tried are:

log: works ✅
status: ✅
plan: does not work ❌
deploy: ❌

Thank you for your help!

$ sqitch --version
sqitch (App::Sqitch) 0.9995

The text was updated successfully, but these errors were encountered:

chamini2 · 2017-03-15T19:29:05Z

My sqitch.conf

[core]
    engine = pg
[target "common"]
    uri = db:pg://sqitch_common@db/db
    registry = sqitch_common
    top_dir = migrations/common
    plan_file = migrations/common/common.plan
    reworked_dir = migrations/common/reworked
[target "ve"]
    uri = db:pg://sqitch_ve@db/db
    registry = sqitch_ve
    top_dir = migrations/country
    plan_file = migrations/country/ve.plan
    reworked_dir = migrations/country/reworked
[target "co"]
    uri = db:pg://sqitch_co@db/db
    registry = sqitch_co
    top_dir = migrations/country
    plan_file = migrations/country/co.plan
    reworked_dir = migrations/country/reworked
[deploy]
    verify = true

Also, may be worth mentioning that the current directory I run these in is a symlink.

theory · 2017-04-21T03:57:10Z

I am not able to replicate this using your sqitch.conf. :-( What output do you see for the plan and deploy commands?

chamini2 · 2017-04-21T16:27:20Z

This is the exact way the directories are:

~/db$ pwd
/home/user/db
~/db$ tree -L 2 -l
.
├── current -> /home/user/db/releases/20170417223451
│   ├── Capfile
│   ├── deploy
│   ├── Gemfile
│   ├── migrations
│   ├── README.md
│   └── REVISION
├── releases
│   ├── 20170412232918
│   ├── 20170412233911
│   ├── 20170412234336
│   ├── 20170412234525
│   └── 20170417223451
└── sqitch.conf

Notice that current is a symbolic link to ./releases/20170417223451
Notice the sqitch.conf in the directory . (current directory)

What output do you see for the plan and deploy commands?

~/db$ cd current
~/db/current$ sqitch plan ve
Plan file sqitch.plan does not exist
~/db/current$ sqitch deploy ve
Plan file sqitch.plan does not exist
~/db/current$ sqitch revert ve
Plan file sqitch.plan does not exist
~/db/current$ sqitch status ve
# On database ve
# Project:  project_name
# Change:   change_sha
# Name:     migration_name
# Deployed: 2017-04-12 01:08:28 +0000
# By:       ...
#
Nothing to deploy (up-to-date)

theory · 2017-04-21T16:32:55Z

You need sqitch.conf in . wherever you run sqitch. Where is your sqitch.plan file?

chamini2 · 2017-04-21T17:26:11Z

When I run the commands it is in .., but it works in my local machine fine.

chamini2 · 2017-04-21T17:30:11Z

Tried either way:

~/db$ ls
current  releases sqitch.conf
~/db$ cp sqitch.conf current/
~/db$ cd current
~/db/current$ ls
Capfile  deploy  Gemfile  migrations  README.md  REVISION  sqitch.conf
~/db/current$ sqitch plan ve
Plan file sqitch.plan does not exist

theory · 2017-04-21T17:34:27Z

So what's happening is it's finding your sqitch.conf, but not your sqitch.plan. Where is the plan file?

chamini2 · 2017-04-21T17:38:45Z

My plan file for that particular target is in

~/db/current/migrations/country$ pwd
/home/user/db/current/migrations/country
~/db/current/migrations/country$ ls
co.plan  deploy  revert  reworked  ve.plan  verify

As stated in the sqitch.conf:

# ...
[target "ve"]
    uri = db:pg://sqitch_ve@db/db
    registry = sqitch_ve
    top_dir = migrations/country
    plan_file = migrations/country/ve.plan
    reworked_dir = migrations/country/reworked
# ...

theory · 2017-04-21T18:06:53Z

So if it's finding sqitch.conf (and it looks like it is), then it tries to find the files as defined there. So if you run sqitch deploy ve, and your current directory does not have the migrations subdirectory, it's not going to find migrations/country/ve.plan.

However, it should work if you're in current, from which it should find migrations/country/ve.plan. I wonder if the symlink is causing problems. Doe it find the plan if you run it from releases/20170417223451 instead of current?

chamini2 · 2017-04-21T22:20:26Z

Does not work either. It's weird.

~$ cd db/releases/20170417223451/
~/db/releases/20170417223451$ ls
Capfile  deploy  Gemfile  migrations  README.md  REVISION  sqitch.conf
~/db/releases/20170417223451$ sqitch status ve
# On database ve
# Project:  project_name
# Change:   change_sha
# Name:     migration_name
# Deployed: 2017-04-12 01:08:28 +0000
# By:       ...
#
Nothing to deploy (up-to-date)
~/db/releases/20170417223451$ sqitch plan ve
Plan file sqitch.plan does not exist

Just in case:

~$ uname -a
Linux staging 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1+deb8u2 (2017-03-07) x86_64 GNU/Linux

theory · 2017-04-21T22:22:12Z

And migrations/country/ve.plan exists there? Mystified…

chamini2 · 2017-04-21T22:25:55Z

Yes, it does:

~/db/releases/20170417223451$ ls migrations/country/
co.plan  deploy  revert  reworked  ve.plan  verify

theory · 2017-04-21T22:27:44Z

Let's do a little debugging. Would you apply this patch and paste the output here?

--- a/lib/App/Sqitch/Plan.pm
+++ b/lib/App/Sqitch/Plan.pm
@@ -127,6 +127,9 @@ sub load {
     my $self = shift;
     my $file = $self->file;
     my $fh = shift || do {
+        say "In ", `pwd`, "Looking for '$file'";
+        say "Absolute path is '", $file->absolute, "'";
+        say "Absolute exists: ", -e $file->absolute ? 'yes' : 'no';
         hurl plan => __x('Plan file {file} does not exist', file => $file)
             unless -e $file;
         hurl plan => __x('Plan file {file} is not a regular file', file => $file)

chamini2 · 2017-04-22T00:42:44Z

Finally did it:

~/db/current$ sqitch status ve
# On database ve
In /home/user/db/releases/20170417223451
Looking for 'migrations/country/ve.plan'
Absolute path is '/home/user/db/releases/20170417223451/migrations/country/ve.plan'
Absolute exists: yes
# Project:  project_name
# Change:   change_sha
# Name:     migration_name
# Deployed: 2017-04-12 01:08:28 +0000
# By:       ...
#
Nothing to deploy (up-to-date)
~/db/current$ sqitch plan ve
In /home/user/db/releases/20170417223451
Looking for 'sqitch.plan'
Absolute path is '/home/user/db/releases/20170417223451/sqitch.plan'
Absolute exists: no
Plan file sqitch.plan does not exist

Same output if I run it from /home/user/db/releases/20170417223451

theory · 2017-04-22T03:25:51Z

WTF. Did you change anything other than the patch, which was just debugging output? Does it work if you run it from current?

chamini2 · 2017-04-22T03:31:07Z

I ran it from current and did not change anything else.

chamini2 · 2017-04-22T03:31:48Z

By the way, this happens in our production and staging machines, so it's probably not a install gone wrong.

theory · 2017-04-22T04:33:20Z

Can you tell what the problem is? Because I sure can't. :-(

chamini2 · 2017-04-22T17:00:45Z

It may be worth mentioning that it works if I set the flag --plan-file

~/db/current$ sqitch --plan-file migrations/country/ve.plan deploy ve
In /home/user/db/releases/20170417223451
Looking for 'migrations/country/ve.plan'
Absolute path is '/home/user/db/releases/20170417223451/migrations/country/ve.plan'
Absolute exists: yes
In /home/user/db/releases/20170417223451
Looking for 'migrations/country/ve.plan'
Absolute path is '/home/user/db/releases/20170417223451/migrations/country/ve.plan'
Absolute exists: yes
Nothing to deploy (up-to-date)

(It still has the patch you asked me to apply)

Without flag

~/db/current$ sqitch deploy ve
In /home/user/db/releases/20170417223451
Looking for 'sqitch.plan'
Absolute path is '/home/user/db/releases/20170417223451/sqitch.plan'
Absolute exists: no
Plan file sqitch.plan does not exist

chamini2 · 2017-04-22T17:17:28Z

I think the problem is more with the sqitch.conf than with the plan file, since if I run the command with any target, it does not throw an error:

~/db/current$ sqitch deploy adsa
In /home/user/db/releases/20170417223451
Looking for 'sqitch.plan'
Absolute path is '/home/user/db/releases/20170417223451/sqitch.plan'
Absolute exists: no
Plan file sqitch.plan does not exist

theory · 2017-04-23T17:01:02Z

That looks like the same error to me.

I do not understand this issue. If the plan file exists at the absolute path and Sqitch says it doesn't, it's…well, weird.

chamini2 · 2017-04-23T17:37:27Z

What I mean is that I think it's not finding the conf, not the plan, so it's falling to a default sqitch.plan file . Is there any patch I could apply to see the configuration it has loaded from the sqitch.conf ?

theory · 2017-04-23T17:47:39Z

No, it's finding the sqitch.conf, because otherwise it wouldn't know where to look for the plan file.

chamini2 · 2017-04-24T13:58:55Z

Ok, testing if it is finding it or not, it did something I did not expect:

~/db$ ls
current  releases  repo  revisions.log  shared  sqitch.conf
~/db$ mv sqitch.conf noconf
~/db$ cd current
~/db/current$ sqitch status ve
# On database ve
In /home/user/db/releases/20170417223451
Looking for 'migrations/country/ve.plan'
Absolute path is '/home/user/db/releases/20170417223451/migrations/country/ve.plan'
Absolute exists: yes
# Project:  project_name
# Change:   change_sha
# Name:     migration_name
# Deployed: 2017-04-12 01:08:28 +0000
# By:       ...
#
Nothing to deploy (up-to-date)
~/db/current$ sqitch deploy ve
In /home/user/db/releases/20170417223451
Looking for 'sqitch.plan'
Absolute path is '/home/user/db/releases/20170417223451/sqitch.plan'
Absolute exists: no
Plan file sqitch.plan does not exist

Even with no sqitch.conf, the status command works (and deploy still does not).

Maybe it's caching the information in the schemas it uses for migrations?

theory · 2017-04-24T16:10:04Z

Ah-ha! Have a look at your ~/.sqitch/sqitch.conf file. I'll bet you've got some stuff in there.

theory · 2017-04-24T16:10:29Z

Or else any sqitch.conf in any parent directories.

chamini2 · 2017-04-24T16:31:41Z

Ok, I do have a ~/.sqitch/sqitch.conf! That is the one I will be using from now on, I will rm the one in ~/db/; either way, the files were identical, so I'm still running into the same problem!

theory · 2017-04-24T17:18:37Z

You should remove all paths to plan files from ~/.sqitch/sqitch.conf; they should only be in the sqitch.conf in your project.

chamini2 · 2017-04-26T17:44:56Z

So sqitch reads both configurations?

theory · 2017-04-26T17:52:43Z

Yes, Sqitch uses the same approach as Git to configuration, even relying on exactly the same format. From the docs:

The file-option can be one of --local, --user, --system, or --file, which specify where the values will be read from or written to. The default is to assume the local config file in the current project directory, for editing, and the all files merged for fetching (see "Files").

It's that last bit that matters: It merges the contents of all the config files when reading config values.

chamini2 · 2017-04-26T18:10:18Z

Ok, trying like this:

~/db/current$ ls
Capfile  deploy  Gemfile  migrations  README.md  REVISION  sqitch.conf

~/db/current$ cat sqitch.conf
[target "common"]
    top_dir = migrations/common
    plan_file = migrations/common/common.plan
    reworked_dir = migrations/common/reworked
[target "ve"]
    top_dir = migrations/country
    plan_file = migrations/country/ve.plan
    reworked_dir = migrations/country/reworked
[target "co"]
    top_dir = migrations/country
    plan_file = migrations/country/co.plan
    reworked_dir = migrations/country/reworked

~/db/current$ cat ~/.sqitch/sqitch.conf
[core]
    engine = pg
[target "common"]
    uri = db:pg://sqitch_common@db/db
    registry = sqitch_common
[target "ve"]
    uri = db:pg://sqitch_ve@db/db
    registry = sqitch_ve
[target "co"]
    uri = db:pg://sqitch_co@db/db
    registry = sqitch_co
[deploy]
    verify = true

~/db/current$ sqitch deploy ve
In /home/user/db/releases/20170424142741
Looking for 'sqitch.plan'
Absolute path is '/home/user/db/releases/20170424142741/sqitch.plan'
Absolute exists: no
Plan file sqitch.plan does not exist

theory · 2017-04-27T00:24:27Z

Yeah, so that's not finding the local sqitch.conf, but looks like it does find /.sqitch/sqitch.conf. Odd. What is the output of these commands:

sqitch target show ve
sqitch config --local target.ve.plan_file

chamini2 · 2017-04-28T00:55:33Z

Output:

~/db/current$ sqitch target show ve
* ve
    URI:           db:pg://sqitch_ve@db/db
    Registry:      sqitch_ve
    Client:        psql
    Top Directory: migrations/country
    Plan File:     migrations/country/ve.plan
    Extension:     sql
    Script Directories:
      Deploy:      migrations/country/deploy
      Revert:      migrations/country/revert
      Verify:      migrations/country/verify
    Reworked Script Directories:
      Reworked:    migrations/country/reworked
      Deploy:      migrations/country/reworked/deploy
      Revert:      migrations/country/reworked/revert
      Verify:      migrations/country/reworked/verify
~/db/current$ sqitch config --local target.ve.plan_file
migrations/country/ve.plan

theory · 2017-07-13T17:48:34Z

Okay, I managed to recreate this issue, I think, although none of the commands worked for me; they all said, “Plan file sqitch.plan does not exist”. Anyway, I believe this patch fixes it:

--- a/lib/App/Sqitch/Command.pm
+++ b/lib/App/Sqitch/Command.pm
@@ -220,7 +220,7 @@ sub parse_args {

     my %engines = map { $_ => 1 } ENGINES;
     for my $arg (@{ $p{args} }) {
-        if ( !$p{no_changes} && $target && $target->plan->contains($arg) ) {
+        if ( !$p{no_changes} && $target && -e $target->plan_file && $target->plan->contains($arg) ) {
             # A change.
             push @{ $rec{changes} } => $arg;
         } elsif ($config->get( key => "target.$arg.uri") || URI->new($arg)->isa('URI::db')) {

I'll figure out how to add a test for this and get it committed. Meanwhile, please let me know if it fixes the issue for you.

chamini2 · 2017-07-13T19:01:59Z

Awesome!

It did fix it for the deploy command, but not for the plan command.

theory · 2017-07-13T19:13:13Z

Confirmed. Hrm…

theory · 2017-07-13T19:17:48Z

Looks like the plan and show commands need --target options, at the very least.

theory · 2017-07-13T22:00:08Z

Okay, 36149b0 fixes plan. Will fix show next.

Ref #324.

theory · 2017-07-13T22:49:22Z

Okay, that should do it, thanks!

chamini2 · 2017-07-14T14:16:15Z

Sure thing, thanks to you!

chamini2 changed the title ~~Some commands finding the conf correctly and some not~~ Some commands not finding the conf correctly Feb 21, 2017

theory self-assigned this Feb 22, 2017

theory added the bug label Feb 22, 2017

theory added this to the v1.0.0 milestone Feb 22, 2017

theory closed this as completed in 48c17b0 Jul 13, 2017

theory reopened this Jul 13, 2017

theory closed this as completed in 36149b0 Jul 13, 2017

theory added a commit that referenced this issue Jul 13, 2017

Add --target option to show.

346d405

Ref #324.

Some commands not finding the conf correctly #324

Some commands not finding the conf correctly #324

Comments

chamini2 commented Feb 21, 2017 • edited

chamini2 commented Mar 15, 2017

theory commented Apr 21, 2017

chamini2 commented Apr 21, 2017

theory commented Apr 21, 2017

chamini2 commented Apr 21, 2017

chamini2 commented Apr 21, 2017

theory commented Apr 21, 2017

chamini2 commented Apr 21, 2017

theory commented Apr 21, 2017

chamini2 commented Apr 21, 2017

theory commented Apr 21, 2017

chamini2 commented Apr 21, 2017

theory commented Apr 21, 2017

chamini2 commented Apr 22, 2017 • edited

theory commented Apr 22, 2017

chamini2 commented Apr 22, 2017

chamini2 commented Apr 22, 2017

theory commented Apr 22, 2017

chamini2 commented Apr 22, 2017 • edited

chamini2 commented Apr 22, 2017 • edited

theory commented Apr 23, 2017

chamini2 commented Apr 23, 2017

theory commented Apr 23, 2017

chamini2 commented Apr 24, 2017

theory commented Apr 24, 2017

theory commented Apr 24, 2017

chamini2 commented Apr 24, 2017

theory commented Apr 24, 2017

chamini2 commented Apr 26, 2017

theory commented Apr 26, 2017

chamini2 commented Apr 26, 2017 • edited

theory commented Apr 27, 2017

chamini2 commented Apr 28, 2017

theory commented Jul 13, 2017

chamini2 commented Jul 13, 2017

theory commented Jul 13, 2017

theory commented Jul 13, 2017

theory commented Jul 13, 2017 • edited

theory commented Jul 13, 2017

chamini2 commented Jul 14, 2017

chamini2 commented Feb 21, 2017 •

edited

chamini2 commented Apr 22, 2017 •

edited

chamini2 commented Apr 22, 2017 •

edited

chamini2 commented Apr 22, 2017 •

edited

chamini2 commented Apr 26, 2017 •

edited

theory commented Jul 13, 2017 •

edited